Methods, apparatus, systems, and storage media for television interaction

Abstract

The application relates to the technical field of intelligent devices, and discloses a method, device and system for television interaction and a storage medium. The method comprises the following steps: in the case that a television is in a fusion interaction mode, acquiring current spatial positioning data, current inertial attitude data and current eye movement tracking data uploaded by a remote controller, and obtaining corresponding current remote controller rough pointing drop point coordinates and current eye movement accurate positioning drop point coordinates; according to the current remote controller rough pointing drop point coordinates and the current eye movement accurate positioning drop point coordinates, iteratively estimating and predicting, based on a multi-source sensor fusion filtering algorithm, to obtain current screen fusion pointing drop point coordinates; in the case that the current eye movement accurate positioning drop point coordinates and the current screen fusion pointing drop point coordinates are matched, if the continuous matching time is greater than a matching time threshold, performing corresponding television interaction according to the current screen fusion pointing drop point coordinates. In this way, the television interaction precision and operation efficiency are effectively improved, and the user television interaction experience is optimized.

Description

Technical Field

[0001] This application relates to the field of smart device technology, such as methods, apparatus, systems, and storage media for television interaction. Background Technology

[0002] Currently, smart TV interaction methods are mainly divided into three categories: traditional physical remote control interaction, independent eye-tracking interaction, and single-space positioning pointing interaction. Traditional infrared and Bluetooth remote controls rely solely on physical buttons or touchpads for cursor control and command confirmation, requiring a two-step "pointing + button" operation for selection. This is cumbersome, lacks natural interaction, and is extremely inefficient in scenarios involving frequent channel switching and menu navigation. Furthermore, the pointing control precision is insufficient to meet the demands of refined screen operation. While remote control solutions based on single-space positioning such as UWB and Bluetooth improve pointing accuracy to some extent, they still rely on physical button confirmation, failing to escape the constraints of traditional interaction. Moreover, single sensors are susceptible to environmental obstructions and electromagnetic interference, resulting in poor positioning stability and robustness. Additionally, related eye-tracking interaction systems often employ an external independent camera architecture on the TV, leading to complex installation, high hardware costs, and time-consuming professional calibration processes, making them difficult for ordinary users to use. Furthermore, fixed eye-tracking acquisition methods are highly demanding in terms of user posture, position, and ambient lighting, easily resulting in recognition failures and point-of-care errors, making them unsuitable for various scenarios.

[0003] In addition, most of the related TV interactive systems have fixed interaction modes and single functions, making it impossible to flexibly switch between multiple interaction methods. They are also difficult to adapt to different lighting and noise environments and user operating habits, lacking the ability to personalize for multiple users. Overall, their interactivity, adaptability, and intelligence are insufficient, failing to meet users' needs for efficient, natural, and diversified TV interaction.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0006] This disclosure provides a method, apparatus, system, and storage medium for television interaction, in order to solve the technical problems of poor pointing accuracy, cumbersome operation, single interaction mode, and insufficient adaptability in existing television interaction methods.

[0007] In some embodiments, the method includes: When the TV is in the converged interactive mode, the system acquires the current spatial positioning data, current inertial attitude data, and current eye-tracking data uploaded by the remote control, and obtains the corresponding coordinates of the current coarse pointing point of the remote control and the coordinates of the current precise eye-tracking point. The spatial positioning data includes one or more of the following: UWB positioning data, Bluetooth AoA positioning data, Wi-Fi positioning data, and infrared positioning data. Based on the current coarse pointing coordinates of the remote control and the current precise eye-tracking positioning coordinates, the current screen fusion pointing coordinates are obtained through iterative estimation using a multi-source sensor fusion filtering algorithm. The multi-source sensor fusion filtering algorithm includes one or more of the following: extended Kalman filter algorithm, particle filter algorithm, neural network fusion algorithm, and fuzzy logic fusion algorithm. If the current eye-tracking precise positioning point coordinates match the current screen fusion pointing point coordinates, and the matching time exceeds the matching time threshold, then the corresponding TV interaction is performed based on the current screen fusion pointing point coordinates.

[0008] In some embodiments, obtaining the corresponding current remote control coarse pointing landing point coordinates includes: Determine the current spatial positioning data and current inertial attitude data, and the corresponding current three-dimensional spatial position coordinates and current three-axis attitude angular coordinates in the world coordinate system of the TV screen; Based on the current three-dimensional spatial position coordinates and the current three-axis attitude angular coordinates, a spatial pointing vector of the remote control is constructed. The spatial pointing vector is then projected onto the two-dimensional screen plane of the TV to obtain the coordinates of the current coarse pointing point of the remote control.

[0009] In some embodiments, obtaining the corresponding current eye-tracking precise positioning point coordinates includes: Based on the current eye-tracking data, extract the raw data of the current user's three-dimensional gaze vector and obtain the corresponding current gaze vector coordinates in the world coordinate system of the TV screen; Based on the personalized eye-tracking calibration parameters matched with the current user, and using the gaze-screen mapping matrix, the coordinates of the current precise eye-tracking positioning point corresponding to the current human eye gaze vector coordinates are obtained.

[0010] In some embodiments, obtaining the coordinates of the current screen fusion pointing point includes: The coordinates of the current coarse pointing point of the remote control and the coordinates of the current precise eye-tracking point are synchronized in time and the coordinate reference is unified. Dynamic weight allocation is then performed to fuse the coordinates to obtain the fused position information. Based on the fused location information, a current system state vector is constructed, which includes the remote controller's spatial position, attitude angle, and screen landing point coordinates. The state is predicted and the observation is corrected by extended Kalman filtering. The current screen fused pointing landing point coordinates are obtained through iterative optimization.

[0011] In some embodiments, the dynamic weight allocation fusion includes: Determine the signal-to-noise ratio and data confidence level of the data source corresponding to the current coarse pointing coordinates of the remote control and the current precise eye-tracking positioning coordinates, respectively; Based on the current ambient light intensity, data source signal-to-noise ratio, and data confidence level, the weight values ​​corresponding to the current remote control coarse pointing landing point coordinates and the current eye-tracking precise positioning landing point coordinates are determined respectively, and the dual landing point adaptive weighted fusion is completed.

[0012] In some embodiments, it also includes: The system identifies the current user and determines the personalized parameters corresponding to the current user. The personalized parameters include one or more of the following: matching time threshold, personalized eye-tracking calibration parameters, weight values, and cursor sensitivity parameters.

[0013] In some embodiments, it also includes: Under the condition that the preset environmental conditions are met or the user operation command is received, determine the current interaction mode of the TV; When the TV is in pure remote control interaction mode, the corresponding TV interaction is performed according to the coordinates of the current coarse pointer of the remote control. When the TV is in pure eye-tracking interaction mode, the corresponding TV interaction is performed based on the precise coordinates of the current eye-tracking location.

[0014] In some embodiments, the apparatus for television interaction includes a processor and a memory storing program instructions, the processor being configured to execute the above-described method for television interaction when executing the program instructions.

[0015] In some embodiments, the device includes: a device body; and a means for television interaction as described above, which is installed on the device body.

[0016] In some embodiments, the storage medium stores program instructions that, when executed, perform the methods described above for television interaction.

[0017] The method, apparatus, system, and storage medium for television interaction provided in this disclosure can achieve the following technical effects: By fusing and correcting multi-type spatial positioning data with eye-tracking data, and combining diverse fusion filtering algorithms, accurate landing point prediction is achieved. This eliminates the traditional two-step button confirmation operation, enabling one-click natural interaction, effectively improving the accuracy and efficiency of TV interaction, and reducing interaction latency and the probability of misoperation. At the same time, it is compatible with multiple positioning and filtering technology solutions, greatly improving the system's environmental adaptability and versatility. It can adapt to different hardware configurations and usage scenarios, overcoming the shortcomings of traditional interaction modes such as singleness, poor adaptability, and insufficient robustness, optimizing the user's TV interaction experience. Furthermore, its strong structural and algorithm adaptability further saves resources.

[0018] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0019] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a schematic diagram of an architecture for a television interactive system provided in an embodiment of this disclosure; Figure 2 This is a flowchart illustrating a television interaction method provided in an embodiment of this disclosure; Figure 3 This is a flowchart illustrating a television interaction method provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of a structure for a television interactive device provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of a structure for a television interactive device provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of a structure for a television interactive device provided in an embodiment of this disclosure; Figure 7 This is a schematic diagram of a television provided in an embodiment of this disclosure. Detailed Implementation

[0020] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0021] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0022] Unless otherwise stated, the term "multiple" means two or more.

[0023] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0024] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0025] In this embodiment, a core technical architecture of multi-source sensor acquisition, multi-modal data fusion, and adaptive mode switching is adopted, which is compatible with multiple spatial positioning methods such as UWB, Bluetooth AoA, Wi-Fi, and infrared, as well as multiple fusion algorithms such as extended Kalman filtering and particle filtering. By integrating inertial attitude acquisition and eye-tracking acquisition modules at the remote control end, terminal spatial positioning data, attitude data, and user eye-tracking data are acquired simultaneously. The coarse pointing landing point and the precise eye-tracking landing point are calculated respectively. Based on the multi-source sensor fusion filtering algorithm, the dual landing point adaptive weighted fusion and iterative correction are completed. Combined with the gaze duration matching mechanism, the implementation is further enhanced. The current buttonless natural confirmation interaction supports adaptive switching between three interaction modes: pure remote control, pure eye tracking, and integrated interaction, as well as multi-user personalized parameter adaptation. This solution effectively solves the technical defects of traditional TV interaction, such as cumbersome operation, low pointing accuracy, poor environmental robustness, high usage threshold, fixed interaction mode, and lack of personalized adaptation. It significantly improves the accuracy, smoothness, and stability of TV interaction, reduces device hardware costs and user calibration operation costs, adapts to various complex usage scenarios such as complex lighting, electromagnetic interference, and changing user postures, and takes into account low power consumption, high adaptability, and strong versatility, significantly optimizing the user's smart TV interaction experience.

[0026] Figure 1 This is a schematic diagram of a structure for a television interactive system provided in an embodiment of this disclosure. Figure 1As shown, the overall architecture of the television interaction system in this embodiment is divided into two main parts: a front-end handheld acquisition terminal and a back-end television processing terminal. Specifically, it includes a remote control 100 and a television 200. The two establish a real-time two-way wireless communication link and work together to complete multi-modal data acquisition, fusion computing, and intelligent interactive control, which can effectively adapt to the intelligent television control needs of multiple scenarios and multiple users. Among them, the remote control 100 is a handheld portable interactive terminal that can be adapted to various handheld device forms such as dedicated television remote controls, intelligent handheld terminals, and smartphones, and has strong versatility. The remote control 100 integrates three core sensing acquisition modules: a spatial positioning acquisition module 110, an inertial attitude acquisition module 120, and an eye-tracking acquisition module 130, to realize the synchronous acquisition of multi-dimensional raw interactive data. Specifically, the spatial positioning acquisition module 110 supports compatible configurations of multiple types of positioning hardware, including one or more of the following: Ultra-Wide Band Positioning Module (UWB), Bluetooth AoA, Wi-Fi, and infrared positioning modules. It can adaptively select the corresponding positioning method according to the hardware configuration and usage environment to achieve accurate calculation of the remote control's spatial position. The inertial attitude acquisition module 12 may include a six-axis or nine-axis inertial measurement unit (IMU), which can collect the remote control's acceleration, gyroscope, and magnetometer data in space in real time and continuously output the remote control's pitch, roll, and yaw three-axis attitude information to provide an attitude reference for screen pointing vector calculation. The eye-tracking acquisition module 130 adopts a miniaturized embedded camera device design, which can be one or more of the following: infrared miniature camera, visible light RGB camera, structured light camera, and event camera. It can collect user eye image data at close range to provide raw data support for subsequent gaze point calculation, user identification, and gaze intent judgment.

[0027] During system operation, the remote controller 100 collects real-time data in parallel through its built-in acquisition modules, continuously acquiring current spatial positioning data, current inertial attitude data, and current user eye-tracking data. It then uploads all raw data to the television 200 in real-time via a preset low-latency wireless communication protocol, completing the front-end data acquisition and transmission process. The television 200, as the core processing and execution terminal of the system, can uniformly receive, time-align, preprocess, and intelligently compute the multi-source heterogeneous data uploaded by the remote controller. Specifically, in some embodiments, the remote controller 100 is configured to collect current UWB positioning data, current inertial attitude data, and current eye-tracking data respectively through the configured spatial positioning acquisition module 110, inertial attitude acquisition module 120, and eye-tracking acquisition module 130, and upload them to the television 200. Television 200 is configured to, when in fusion interaction mode, acquire current spatial positioning data, current inertial attitude data, and current eye-tracking data uploaded by remote control 100, and obtain the corresponding current remote control coarse pointing landing point coordinates and current eye-tracking precise positioning landing point coordinates; based on the current remote control coarse pointing landing point coordinates and current eye-tracking precise positioning landing point coordinates, iteratively predict the current screen fusion pointing landing point coordinates using a multi-source sensor fusion filtering algorithm; if the current eye-tracking precise positioning landing point coordinates match the current screen fusion pointing landing point coordinates, and if the matching time exceeds a matching time threshold, perform corresponding television interaction based on the current screen fusion pointing landing point coordinates. The multi-source sensor fusion filtering algorithm includes one or more of the following: extended Kalman filter algorithm, particle filter algorithm, neural network fusion algorithm, and fuzzy logic fusion algorithm.

[0028] Specifically, when the TV 200 is in fusion interactive mode, it jointly calculates the received spatial positioning data and inertial attitude data to construct the remote control's spatial pointing vector and projects it onto the TV screen, obtaining the coarse coordinates of the remote control's pointing point. Simultaneously, based on received eye-tracking data, combined with user-personalized calibration parameters and a gaze mapping matrix, it calculates the high-precision coordinates of the user's eye-tracking screen pointing point. The TV 200's built-in data fusion processing unit is equipped with a multi-source sensor fusion filtering algorithm, specifically including one or more of the extended Kalman filter, particle filter, neural network fusion algorithm, and fuzzy logic fusion algorithm. This algorithm can adaptively allocate weights based on the environmental signal-to-noise ratio and data confidence level, iteratively optimizing and fusion correcting the dual-path pointing point coordinates to output more stable and accurate current screen fusion pointing point coordinates. The TV 200 system compares the real-time eye-tracking coordinates with the coordinates of the screen's fused pointing point, continuously monitoring the matching duration. When the duration of this matching exceeds a preset threshold, it determines that the user has generated a valid interaction intent and automatically executes corresponding TV interaction operations such as clicking, menu switching, channel adjustment, and parameter setting based on the final screen fused pointing point coordinates. This system, through multi-sensor integration on the remote control and intelligent fusion computing on the TV, overcomes the limitations of traditional single-interaction modes and avoids the drawbacks of complex installation, cumbersome calibration, and poor stability of external eye-tracking devices. It achieves a high-precision, low-barrier, and multi-scenario adaptable intelligent TV interaction effect.

[0029] Based on the aforementioned system for television interaction, the television can perform corresponding intelligent interactive operations, relying on multi-source sensor data acquisition and fusion processing logic to complete automated interactive control. Figure 2 This is a flowchart illustrating a television interaction method provided in an embodiment of this disclosure. Figure 2 As shown, the process for television interaction includes: Step 201: When the TV is in the converged interactive mode, obtain the current spatial positioning data, current inertial attitude data, and current eye-tracking data uploaded by the remote control, and obtain the corresponding coordinates of the current coarse pointing point of the remote control and the coordinates of the current precise eye-tracking point. The spatial positioning data includes one or more of the following: UWB positioning data, Bluetooth AoA positioning data, Wi-Fi positioning data, and infrared positioning data.

[0030] In this embodiment of the disclosure, after the television is powered on, it may default to the converged interactive mode, or the television may enter the converged interactive mode according to user operation instructions, or it may automatically enter the converged interactive mode according to environmental conditions. Alternatively, in some embodiments, the television may only have one mode, which is the converged interactive mode.

[0031] The remote control can flexibly acquire spatial positioning data through various hardware methods, exhibiting strong hardware adaptability: it can collect high-precision spatial positioning data through a built-in ultra-wideband (UWB) positioning module, or individually collect corresponding positioning data through Bluetooth AoA, Wi-Fi, and infrared positioning modules. Furthermore, it can combine multiple positioning modules to simultaneously collect data based on environmental interference and hardware configuration, achieving redundancy backup and precision complementarity. Therefore, the spatial positioning data in this embodiment is compatible with one or more of ultra-wideband (UWB), Bluetooth AoA, Wi-Fi, and infrared positioning data. Similarly, the eye-tracking acquisition module on the remote control is compatible with various acquisition hardware such as infrared miniature cameras, visible light RGB cameras, structured light cameras, and event cameras. It can adaptively select the corresponding acquisition method based on ambient light intensity and the user's device wearing status, stably acquiring user eye images and parsing them to obtain high-precision current eye-tracking data, providing data support for subsequent gaze point calculation.

[0032] The television can communicate with the remote control to acquire multimodal sensor data collected by the remote control. The data corresponding to the current moment may include: current spatial positioning data, current inertial attitude data, and current eye-tracking data. After acquiring the multimodal sensor data, the television can independently perform calculations and analyses on each type of sensor data, ultimately outputting the corresponding coordinates of the current coarse remote control pointing point and the current precise eye-tracking positioning point.

[0033] In some embodiments, obtaining the corresponding current remote control coarse pointing landing point coordinates includes: determining the current spatial positioning data and the current inertial attitude data, the current three-dimensional spatial position coordinates and the current three-axis attitude angular coordinates in the world coordinate system of the TV screen; constructing a remote control spatial pointing vector based on the current three-dimensional spatial position coordinates and the current three-axis attitude angular coordinates; projecting the spatial pointing vector onto the two-dimensional screen plane of the TV to obtain the current remote control coarse pointing landing point coordinates.

[0034] Taking the current spatial positioning data as an example, and using UWB positioning data as an example, the TV device pre-deploys four sets of rectangularly distributed, clock-synchronized UWB anchor point arrays. The coordinates of the four UWB anchor points are pre-calibrated and fixed in the world coordinate system of the TV screen, forming a stable global positioning benchmark, providing a coordinate reference benchmark for the remote control's spatial position calculation. The UWB positioning module on the remote control can transmit high-frequency narrow-pulse UWB signals in real time, possessing the characteristics of anti-multipath interference, strong penetration, and high positioning accuracy. The four UWB anchor points on the TV can synchronously receive the UWB pulse signals transmitted by the remote control and independently record the precise timestamp of the signal arriving at their respective anchor points. After the TV main control unit collects the signal arrival time data from four anchor points, it obtains the signal arrival time difference (TDOA) between different anchor points through differential calculation. Based on the TDOA hyperbolic positioning algorithm principle, a set of positioning equations is constructed. Specifically, for any point on the remote control, the signal transmission distance difference to two different anchor points is constant, corresponding to a fixed time difference. By solving multiple sets of hyperbolic equations, the real-time three-dimensional spatial coordinates of the remote control in the TV screen's world coordinate system can be accurately calculated, precisely locking the spatial distribution position of the remote control. Let the preset fixed coordinates of the four UWB anchor points on the TV end be (x i y i , z i) The coordinates of the remote control are (x, y, z), i = 1, 2, 3, 4, and Δt is the real-time spatial coordinates of the remote control. ij Let be the time difference between the arrival of the signal at the i-th anchor point and the j-th anchor point, and let c be the speed of light. Then the distance difference satisfies the hyperbolic positioning equation. - The three-dimensional spatial coordinates of the remote controller can be stably solved by solving multiple sets of equations simultaneously.

[0035] Based on the spatial position calculation, the TV synchronously receives the IMU inertial attitude data uploaded in real time by the remote control. Through filtering and noise reduction to eliminate abnormal data such as jitter and drift, the TV accurately calculates the remote control's current three-axis attitude angular coordinates, fully covering pitch, roll, and yaw angles, precisely representing the remote control's handheld deflection attitude and spatial orientation. Combining the calculated three-dimensional spatial position coordinates and three-axis attitude angular coordinates, the TV can accurately construct the three-dimensional spatial pointing vector of the remote control pointing at the TV screen, completing the spatial dimension pointing model. Finally, based on preset screen size parameters, screen resolution parameters, and coordinate mapping rules, the TV accurately projects and maps the three-dimensional spatial pointing vector onto the two-dimensional screen plane, and completes coordinate normalization and correction processing, ultimately outputting stable and reliable coarse coordinates of the current remote control's pointing point. This UWB and IMU joint calculation method can achieve centimeter-level coarse positioning results, has strong resistance to environmental obstruction and electromagnetic interference, and can provide stable coarse positioning reference data for subsequent multi-source data fusion, effectively compensating for the shortcomings of single eye tracking which is susceptible to the influence of lighting and user posture.

[0036] In some embodiments, obtaining the corresponding current eye-tracking precise positioning point coordinates includes: extracting the original data of the current user's three-dimensional gaze vector based on the current eye-tracking data, and obtaining the current gaze vector coordinates in the world coordinate system of the TV screen; and obtaining the current eye-tracking precise positioning point coordinates corresponding to the current gaze vector coordinates based on the gaze screen mapping matrix according to the personalized eye-tracking calibration parameters matched with the current user.

[0037] The remote control is equipped with a miniature eye-tracking acquisition device, which can use any of the following: an infrared miniature camera, a visible light RGB camera, a structured light camera, or an event camera. It can acquire real-time image data of the user's eyes at close range and at a high frame rate. Relying on the built-in lightweight image processing unit, it completes eye image preprocessing, including image grayscale conversion, filtering and noise reduction, image enhancement, and ROI eye region cropping. It effectively eliminates abnormal factors such as ambient light interference, eye blinking artifacts, and image noise, and outputs stable and reliable current eye tracking data. After acquiring the current eye-tracking data uploaded by the remote control, the TV performs key feature detection and extraction on the preprocessed eye image to accurately identify core eye features such as the center position of the user's pupil and corneal reflection feature points. Based on this, the TV calculates the raw three-dimensional gaze vector data of the current user's eyes. To achieve coordinate unification with the spatial positioning data, the TV performs coordinate system transformation and alignment correction on the raw gaze vector data, mapping the gaze vector in the camera's local coordinate system to the TV screen's world coordinate system, obtaining standardized and unified current gaze vector coordinates, and completing the quantitative modeling of the user's gaze direction. Based on the calculation of the human eye's line of sight vector coordinates, the TV terminal retrieves personalized eye-tracking calibration parameters that uniquely match the currently logged-in user. These parameters are obtained through initial rapid calibration and iterative updates through implicit learning during daily use, adapting to individual differences in eye physiological characteristics, viewing distance, line of sight deviation habits, and wearing glasses among different users, effectively eliminating systematic errors caused by general parameters. The television pre-stores a trained gaze-to-screen mapping matrix, which establishes a precise correspondence between the human eye's three-dimensional gaze vector and the television's two-dimensional screen pixel coordinates. Based on the retrieved user-personalized eye-tracking calibration parameters, the current human eye gaze vector coordinates are corrected and compensated. Then, the matrix transformation and coordinate mapping operations are performed using the pre-set gaze-to-screen mapping matrix, ultimately providing a precise solution for the corresponding current eye-tracking location coordinates. This eye-tracking localization solution relies on user-specific calibration parameters and a gaze mapping model to achieve sub-pixel-level high-precision localization of screen landing points. It compensates for the insufficient accuracy of single UWB coarse localization and complements the accuracy of UWB and IMU coarse localization data. This provides high-precision and high-reliability fine localization reference data for subsequent multi-source sensor data fusion, landing point matching verification, and intelligent interaction confirmation.

[0038] Step 202: Based on the current coarse pointing coordinates of the remote control and the current precise eye-tracking pointing coordinates, the current screen fusion pointing coordinates are obtained by iterative estimation using a multi-source sensor fusion filtering algorithm. The multi-source sensor fusion filtering algorithm includes one or more of the following: extended Kalman filter algorithm, particle filter algorithm, neural network fusion algorithm, and fuzzy logic fusion algorithm.

[0039] In this embodiment, high-precision fusion of dual-landing point data can be achieved through multi-source data preprocessing, adaptive weighted fusion, and iterative filtering optimization. Specifically, this includes three core processes: data alignment and unification, dynamic weight fusion, and state iterative optimization. In some embodiments, obtaining the current screen fusion pointing landing point coordinates includes: performing time synchronization alignment and coordinate benchmark unification on the current remote control coarse pointing landing point coordinates and the current eye-tracking precise positioning landing point coordinates, and then performing dynamic weight allocation fusion to obtain fused position information; based on the fused position information, constructing a current system state vector containing the remote control's spatial position, attitude angle, and screen landing point coordinates; performing state prediction and observation correction through extended Kalman filtering; and iteratively optimizing to obtain the current screen fusion pointing landing point coordinates.

[0040] The television can first perform preprocessing on the coarse coordinates of the current remote control pointing point and the precise coordinates of the current eye-tracking positioning point, completing the time synchronization and alignment of the two heterogeneous data streams and the unified processing of the coordinate reference. Similarly, taking the current spatial positioning data as the current UWB positioning data as an example, since there are differences in the acquisition frame rate and output timing between the UWB and IMU coarse positioning data and the eye-tracking fine positioning data, the system achieves precise time alignment of the two data streams through timestamp interpolation compensation and timing calibration mechanisms. At the same time, it unifies the two types of point coordinates into the two-dimensional pixel coordinate system of the television screen, eliminating fusion errors caused by inconsistent coordinate dimensions and timing misalignments, and providing a standardized data source for subsequent data fusion calculations. After data alignment and benchmark unification are completed, dynamic weight allocation fusion can be performed. In some embodiments, dynamic weight allocation fusion includes: determining the data source signal-to-noise ratio and data confidence level corresponding to the current remote control coarse pointing landing point coordinates and the current eye-tracking precise positioning landing point coordinates, respectively; and determining the weight values ​​corresponding to the current remote control coarse pointing landing point coordinates and the current eye-tracking precise positioning landing point coordinates based on the current ambient light intensity, data source signal-to-noise ratio, and data confidence level, thereby completing the dual-landing point adaptive weighted fusion.

[0041] The TV uses real-time quantization to calculate the signal-to-noise ratio (SNR) and data confidence level of the data sources corresponding to the current remote control coarse pointing coordinates and the current eye-tracking precise positioning coordinates. The SNR represents the effective signal ratio and anti-interference capability of the current sensor data, while the data confidence level represents the reliability and stability of the current coordinate calculation results. Based on this, the TV can also dynamically evaluate the real-time effectiveness and reliability of the two types of data by combining real-time collected environmental parameters such as ambient light intensity and spatial electromagnetic interference intensity: when the lighting is sufficient and eye recognition is stable, the weight of the eye-tracking precise positioning data is increased; when the lighting is dim, eye recognition is easily interfered with, or electromagnetic interference is strong, and UWB positioning fluctuations are small, the weight of the UWB coarse positioning data is adaptively increased. Finally, the optimal dynamic weight values ​​corresponding to the two landing coordinates are determined, and the adaptive weighted calculation of the dual landing data is completed according to the weighted fusion formula, outputting the initial fused position information to achieve complementary advantages and error suppression. After obtaining the initial fusion position information, the TV can construct a complete current system state vector. This state vector integrates the remote control's real-time spatial position parameters, three-axis attitude angle parameters, and the initial fusion landing point coordinates of the screen, fully representing the global operating state of the current interactive system. For example, the current system state vector... Where (x, y, z) are the control position parameters, and ( Let Z be the attitude angle parameter, and the observation equation be Z = Hx + V, where Z is the observation vector, H is the observation matrix, and V is the observation noise.

[0042] Then, iterative optimization can be achieved by adapting to various fusion filtering algorithms. For example, the extended Kalman filter algorithm can be used as the core fusion strategy, while also being compatible with various implementation methods such as particle filter algorithm, neural network fusion algorithm, and fuzzy logic fusion algorithm. It can adaptively switch according to the computing power of the device and the complexity of the environment. Through the state prediction, observation update, and error correction iterative mechanism of the extended Kalman filter, the initial fusion position information is iteratively estimated and biased, continuously eliminating single-point abnormal data, compensating for the steady-state error and dynamic drift error of the system, and continuously optimizing the accuracy and stability of the screen landing point coordinates. Finally, iterative convergence yields the current screen fusion pointing landing point coordinates with high accuracy and high robustness. This fusion process dynamically adjusts the weights across three dimensions—environment, signal-to-noise ratio, and confidence level—combined with iterative filtering optimization. This completely resolves the shortcomings of single UWB positioning, which is susceptible to electromagnetic interference, and single eye-tracking positioning, which is susceptible to the influence of lighting and user posture. It balances the stability of coarse positioning with the high accuracy of fine positioning, so that the final fused landing point coordinates have the advantages of low drift, high accuracy, and strong anti-interference. This provides a reliable core coordinate basis for subsequent gaze matching determination and automated TV interaction.

[0043] Step 203: If the current eye-tracking precise positioning point coordinates match the current screen fusion pointing point coordinates, and the matching time is longer than the matching time threshold, perform the corresponding TV interaction based on the current screen fusion pointing point coordinates.

[0044] In some embodiments, intent determination and interaction execution are also required. The TV can rely on a dual-coordinate matching verification and timing monitoring mechanism to achieve buttonless intelligent interaction confirmation, completely abandoning the traditional two-step operation mode of "pointing and physical buttons," and improving the naturalness of interaction and operation efficiency. After iteratively solving for the coordinates of the current screen fusion pointing point, the TV performs real-time overlap matching verification between the current eye-tracking precise positioning point coordinates and the current screen fusion pointing point coordinates. The coordinate deviation threshold is used to determine whether the two are in a valid matching state. When the deviation between the two sets of high-precision pointing point coordinates is less than the preset deviation threshold, it is determined that the current gaze point and the remote control pointing point are validly matched, indicating that the user's gaze is focused on the target interaction area. Furthermore, the TV also features continuous matching time monitoring, which tracks the duration of continuous matching across the two coordinates in real time. When the duration exceeds the system's preset matching time threshold, it confirms that the user has a stable and effective interactive selection intent, eliminating interference from minor eye movements or unintentional glances. Ultimately, based on the highly accurate and robust current screen fusion pointing coordinates, it automatically executes the corresponding TV interactive operation, including menu selection, channel switching, function activation, parameter confirmation, cursor locking, and other touch interaction commands, achieving a seamless and natural interactive experience of "point, look, select." The matching time threshold is a personalized and adjustable parameter adapted to user habits, and can be 1 second, 3 seconds, or 5 seconds, etc. It can be adaptively configured according to different users' operation speed and usage preferences, balancing operational sensitivity and anti-accidental touch capabilities. In this embodiment, based on the preferred implementation of default gaze duration confirmation, multiple equivalent confirmation alternatives are compatible, and can be flexibly switched according to user settings and scenario conditions, further broadening the interactive adaptation scenarios. Specifically, these may include: a blink confirmation method, which accurately identifies the user's specific blinking pattern through eye image sequences, preferentially using double blinking features as an effective confirmation command, and distinguishing between natural blinks and active confirmation blinks through algorithms, eliminating the need to wait for gaze duration and resulting in a more natural and efficient interactive response; a remote control gesture confirmation method, which relies on the remote control's IMU posture data to recognize preset motion gestures in real time, such as horizontal swaying and fixed-point pitching, providing intuitive operation feedback and high interaction recognition, suitable for users who prefer manual assisted confirmation; and a voice confirmation method, which uses the TV or remote control's built-in voice recognition module to parse the user's short confirmation voice commands to trigger the interaction, eliminating the need for hand-eye coordination and suitable for multi-tasking scenarios such as two-handed operation or focused movie watching. These are just a few examples.

[0045] Here, stable and reliable automated interactive confirmation is achieved through precise matching of dual coordinates and duration threshold verification. It is also compatible with multiple confirmation methods, solving the problems of cumbersome and rigid modes in traditional TV interactive operations. It takes into account the naturalness, accuracy, and anti-accidental touch capabilities of interaction, and adapts to the diversified interactive needs of different users, different environments, and different operation tasks.

[0046] As can be seen, in this embodiment, the TV interaction method relies on a complete technical chain of multimodal sensor data collaborative acquisition, coarse and fine positioning complementary solution, dynamic weighted multi-source fusion filtering, and temporal intent verification. It achieves stable coarse remote control pointing positioning through UWB and IMU, and completes high-precision gaze point calculation by combining a refined eye-tracking algorithm. Then, through environmental adaptive dynamic weighted fusion and extended Kalman filtering iterative optimization, it effectively integrates the advantages of the two types of positioning data, suppresses various environmental interferences and system errors, and finally accurately identifies the user's interaction intent based on a dual-point matching duration determination mechanism, achieving natural smart TV interaction without physical buttons. This disclosed embodiment solves the technical pain points of traditional TV interaction, such as cumbersome operation, single positioning accuracy, weak environmental robustness, high false trigger rate, and fixed interaction modes. It is also compatible with multiple confirmation methods such as blinking, gestures, and voice, as well as multi-dimensional intelligent switching modes based on scenarios, tasks, and physiological states. It can adapt to different lighting conditions, electromagnetic environments, user operating habits, and usage scenarios, balancing interaction accuracy, stability, naturalness, and versatility, significantly improving the intelligence level of smart TV human-computer interaction and user experience.

[0047] Of course, different users correspond to different personalized parameters. Therefore, in some embodiments, the method further includes: identifying the current user and determining the personalized parameters corresponding to the current user. The personalized parameters include one or more of the following: matching time threshold, personalized eye-tracking calibration parameters, weight values, and cursor sensitivity parameters.

[0048] In some embodiments, a multi-dimensional user identification and adaptive personalized parameter iteration mechanism can be adopted. This mechanism relies on both biometrics and operational behavior characteristics to differentiate users, and combines machine learning algorithms to achieve accurate user classification and parameter adaptation. The overall identification process includes four core steps: eye-tracking feature extraction, operational habit analysis, machine learning classification, and adaptive parameter updates. First, eye-tracking feature extraction is performed. Based on continuous frame eye-tracking data, the system stably extracts unique biometrics such as the user's gaze distribution pattern, gaze saccade speed, and dynamic changes in pupil size, forming the core feature vector for user identification. Second, operational habit analysis is conducted. Long-term statistical recording of user daily interaction speed, common interaction preference patterns, confirmation methods, and typical viewing distances forms a user behavior feature database. Then, a support vector machine or neural network machine learning model is used to fuse and classify eye-tracking biometrics and operational behavior characteristics, accurately completing the current user identification and matching. Finally, various personalized parameters are dynamically iteratively corrected based on real-time user usage data to achieve adaptive parameter optimization and adapt to changes in long-term user habits. Of course, the configured personalized parameter system can fully adapt to the entire algorithm process of user TV interaction, specifically covering three types of core parameters: First, calibration parameters, mainly pupil-screen gaze mapping parameters, are used to correct gaze positioning deviations caused by physiological differences in the eyes of different users; second, core interaction parameters, including key parameters such as landing point matching time threshold, dual data source dynamic weight values, and cursor movement sensitivity, which directly determine interaction accuracy, anti-accidental touch capability, and response speed; and third, user preference setting parameters, including custom configurations such as default interaction mode and interface operation theme, to fully meet users' personalized interaction needs. Meanwhile, this solution is equipped with multiple alternative user identification algorithms, which can be flexibly switched according to hardware configuration, usage scenario, and privacy requirements, improving system adaptability and scalability. The first is a recognition scheme based on operating habits, which relies on user operating speed, mode selection preferences, confirmation operation time, and other behavioral characteristics to complete identity recognition. It does not require collecting eye biometric features, offering stronger privacy protection and is suitable for application scenarios with high data privacy requirements. Its only drawback is a longer recognition learning cycle and slightly lower initial accuracy. The second is a voice-based recognition scheme, which uses the device's built-in microphone to collect the user's unique voice features and the operating environment's sound features to complete identity verification. It requires no additional hardware and can reuse existing voice interaction modules, making it compatible with TV systems with built-in voice interaction functions. Its recognition accuracy may be somewhat affected in noisy environments. The third is a grip posture-based recognition scheme, which relies on the remote control's pressure sensor array to collect the user's grip posture, contact surface pressure distribution, and other non-sensory features to achieve seamless automatic identity recognition. The interactive experience is natural and smooth, requiring no user-triggered recognition action. It is suitable for high-end smart TV products that pursue the ultimate interactive experience, but its only drawback is relatively high hardware costs. Through the aforementioned user identification and personalized adaptation mechanism, this system can achieve independent parameter storage for multiple users, accurate identity differentiation, and dynamic parameter adaptive updates. It adapts the interaction logic differently for different users based on their eye characteristics, operation speed, usage habits, and privacy needs, completely solving the problems of poor experience consistency, low adaptability, and insufficient fault tolerance caused by the uniform parameter adaptation of traditional TV interaction for all users. This further enhances the humanization, intelligence, and scene adaptation capabilities of the entire multimodal intelligent interaction system.

[0049] During television interaction, the television can operate in multiple modes, including: integrated interaction mode, pure remote control interaction mode, and pure eye-tracking interaction mode. In integrated interaction mode, the television can perform corresponding interactions based on the coordinates of the current screen's fused pointing point. In pure remote control interaction mode, the television can perform corresponding interactions based on the coordinates of the current remote control's coarse pointing point. In pure eye-tracking interaction mode, the television can perform corresponding interactions based on the precise coordinates of the current eye movement. In some embodiments, this method also includes a mode adaptive switching and differentiated interaction execution process: When preset environmental conditions, task conditions, user physiological state conditions, or user operation commands are met, the current interaction mode of the television is determined; in pure remote control interaction mode, corresponding television interactions are performed based on the coordinates of the current remote control's coarse pointing point; in pure eye-tracking interaction mode, corresponding television interactions are performed based on the precise coordinates of the current eye movement; and in integrated interaction mode, high-precision intelligent interaction is achieved based on the coordinates of the screen's fused pointing point after multi-source data fusion. The determination and switching of TV interaction modes can be comprehensively determined by combining real-time environmental conditions, user operation instructions, current interaction task type and user physiological state, so as to achieve adaptive intelligent switching in multiple scenarios and working conditions. Specifically, different interaction modes are adapted to different usage scenarios and operational needs. The hardware working state, computing logic, and interaction mechanism of each mode are independent and can be started and stopped as needed. Among them, the pure remote control interaction mode is suitable for scenarios with low ambient light, poor eye-tracking camera imaging quality that cannot work properly, users wearing glasses or contact lenses that reduce eye-tracking accuracy, or users who prefer traditional remote control operation. In this mode, only the UWB positioning module and IMU inertial attitude module work normally. The eye-tracking camera is automatically turned off or enters a low-power sleep state. The TV can calculate the coordinates of the coarse pointing point of the remote control based solely on UWB and IMU data. Users can complete the interaction confirmation through traditional methods such as physical buttons on the remote control or long press touch. The cursor follows the coarse pointing point and is displayed on the TV screen in real time. This mode has the lowest overall hardware power consumption, perfectly compatibility with users' traditional operating habits, and can ensure the availability of interactive functions in extreme environments. Only the positioning accuracy is relatively limited. The pure eye-tracking interaction mode is suitable for special scenarios where users hold a remote control but do not need to point, require precise selection of small targets on the screen, or have limited hand movement. In this mode, the eye-tracking camera operates at high frequency, and the UWB positioning module automatically enters a low-power energy-saving state. The system relies on eye-tracking algorithms to calculate the user's gaze point in real time throughout the process, and determines the user's interaction intent by using a gaze duration threshold. When the effective gaze duration exceeds a preset threshold, the interaction is automatically confirmed, and the cursor is directly and accurately displayed at the user's gaze location, without the need for hand assistance. This mode offers the most natural operation, requiring no manual control and adapting to refined interface selection operations. It only has certain adaptation requirements regarding the user's sitting position, head posture, and ambient lighting conditions. In this embodiment, the fusion interaction mode of remote control pointing and eye-tracking confirmation can be the default regular working mode, adaptable to most daily movie watching, interface operation, and function setting scenarios, especially suitable for scenarios requiring quick and accurate operation selection and for new users learning to use the system. In this mode, the UWB positioning module and the eye-tracking camera work synchronously and collaboratively. The UWB and IMU output stable coarse pointing points, while the eye-tracking algorithm outputs high-precision positioning points. The system completes dual-point optimization fusion through a multi-source fusion algorithm. When the system detects that the user's gaze point matches the remote control pointing point for a specified duration, it automatically triggers interactive confirmation, and can simultaneously output visual, auditory, and other multimodal feedback prompts. This mode combines high operational efficiency with high positioning accuracy, resulting in the optimal overall interactive experience. This disclosure supports a multi-dimensional and multi-form intelligent mode switching mechanism, including three core switching methods: manual switching, automatic switching, and hybrid switching, ensuring flexible and smooth mode switching. The manual switching method allows users to actively switch interaction modes via preset gestures or voice control commands. The automatic switching method relies on multi-sensor environmental perception and status monitoring to automatically match the optimal interaction mode based on real-time ambient light intensity, spatial noise level, electromagnetic interference intensity, usage time, user operating habits, fatigue level, attention state, and current browsing, selection, and input task requirements, without requiring manual user intervention. The hybrid switching method relies on user-defined priority settings; based on user-defined preferences, the TV automatically fine-tunes and adapts the interaction mode according to real-time operating conditions, balancing user subjective preferences with objective needs of the scenario. Meanwhile, multiple mode switching alternative optimization schemes are also configured to further broaden the scope of scene adaptation and enhance the level of intelligence. The first is a scene-based automatic switching scheme, which relies on multiple sensors to collect scene data such as ambient light, noise, and time of day in real time, achieving fully automated mode switching without user intervention. This is suitable for regular daily scenarios and solves the problem of tedious manual switching. The second is a task-based intelligent switching scheme, which uses TV interface context analysis technology to identify the current interaction task type and match the optimal interaction mode accordingly, effectively improving the interaction efficiency under different operation tasks and adapting to refined and complex operation scenarios. The third is a biosignal-based intelligent switching scheme, which can be expanded to integrate biosensors such as heart rate and skin conductance to identify the user's fatigue level and attention level, adaptively adjusting the interaction mode and parameters according to the user's real-time physiological state, adapting to high-end intelligent interaction scenarios. To ensure continuity of mode switching and consistency of user experience, a robust smooth switching mechanism can be set up. During mode switching, data such as the current cursor position, interaction state, and parameter configuration are automatically cached, achieving seamless state saving and continuation. Simultaneously, feedback methods such as interface transition animations, slight vibrations of the remote control, and short prompt sounds are used to achieve smooth mode transitions and clear user prompts, completely avoiding issues such as mode switching stutters, state loss, and disjointed experiences. Through the aforementioned multi-mode layered adaptation and intelligent switching mechanism, the embodiments of this disclosure can dynamically adapt to complex environments, diverse tasks, and differentiated user states, while also considering the interaction requirements of low power consumption, high precision, and high naturalness. This solves the shortcomings of traditional TV interaction modes, limited scene adaptation, and inability to adaptively optimize, significantly improving the system's versatility and intelligence.

[0050] Of course, the power consumption issues that may arise from multimodal sensor collaboration and parallel computation of multiple algorithms can be addressed in some embodiments by building a comprehensive and hierarchical intelligent power consumption optimization system. This system can achieve refined power consumption control from multiple dimensions, including hardware module scheduling, dynamic adjustment of sampling frequency, algorithm lightweighting, data transmission optimization, and intelligent wake-up prediction. Under the premise of ensuring interaction accuracy and real-time response, it can minimize the overall power consumption of the device and improve its battery life and long-term operational stability. Specifically, power consumption optimization measures include five core strategies: First, a dynamic frequency adjustment strategy. The system can adaptively adjust the sampling frequency of each sensor module according to the current interaction task intensity, user operation status, and environmental interference level. When the user is stationary and the environment is stable, the sampling frame rate of UWB positioning, IMU attitude acquisition, and eye-tracking image acquisition is automatically reduced to reduce the ineffective power consumption caused by high-frequency sampling. When user operation, eye movement, or remote control posture change is detected, high frame rate sampling is immediately restored to ensure smooth interaction and positioning accuracy, achieving a dynamic balance between performance and power consumption. Second, a module-level independent power management strategy. The system supports independent sleep and wake-up scheduling for each sensor module. Hardware modules can be started and stopped as needed according to the current interaction mode. Non-working modules automatically enter a low-power sleep state. For example, the eye-tracking camera sleeps in pure remote control mode, and the UWB module sleeps in pure eye-tracking mode, preventing power waste from long-term idle operation of multiple modules. Thirdly, there is a lightweight algorithm optimization strategy. This strategy adaptively switches between low-complexity algorithms based on device computing power and interaction scenarios. In routine daily interaction scenarios, lightweight filtering and feature tracking algorithms are prioritized to simplify iterative calculations, reduce the computing power overhead of the main control chip, and decrease power consumption. In high-precision scenarios, high-precision fusion algorithms are enabled to balance interaction performance with computing power and power consumption. Fourthly, there is a data compression and transmission strategy. The remote control performs lightweight compression encoding on real-time collected spatial positioning data, posture data, and eye-tracking image data, removing redundant frames, duplicate data, and invalid noise data to reduce the amount of data transmitted via wireless communication, effectively reducing wireless radio frequency transmission power consumption and improving communication energy efficiency. Fifthly, there is a predictive intelligent wake-up strategy. Based on long-term recorded user usage time periods, operating habits, and viewing patterns, the system builds a user behavior prediction model. This model can predict the user's power-on usage time in advance and complete device pre-wake-up and parameter initialization before user use, avoiding user operation delays and mitigating the problem of long-term high-frequency standby power consumption. Based on the above optimization strategies, the TV can achieve precise hierarchical power consumption control in various working states, forming standardized power consumption allocation indicators: the static current of the whole machine in standby mode is less than 10μA, achieving ultra-low static power consumption; the working current in pure remote control interaction mode is less than 5mA, achieving optimal power saving effect based on a minimalist hardware architecture; the working current in pure eye-tracking interaction mode is less than 8mA, adapting to refined low-power interaction scenarios; in the integrated interaction mode, multiple modules work together, and the working current is stably controlled within 12mA, meeting the needs of daily high-frequency use; the short-term peak working current is strictly controlled within 50mA, which can effectively avoid the problem of device overheating and sudden drop in battery life caused by excessive instantaneous power consumption. The overall power consumption performance is far superior to traditional multimodal interactive devices, and it has excellent energy-saving and battery life capabilities.

[0051] In some embodiments, to ensure the long-term high-precision and high-stability operation of the multimodal interaction system, this solution establishes a full lifecycle calibration and maintenance mechanism that includes initial calibration, online dynamic calibration, and multi-user refined management. This solves the shortcomings of traditional TV eye-tracking interaction calibration processes, such as cumbersome calibration procedures, long calibration time, easy parameter drift, poor multi-user adaptability, and high maintenance costs, and enables rapid equipment deployment, long-term self-calibration, and multi-user adaptive maintenance.

[0052] The initial calibration process is performed after the device is first used or the parameters are reset. The overall calibration process is streamlined and efficient, with the total calibration time controlled within 2 minutes. Compared with the 5-10 minute calibration time of traditional eye-tracking interaction devices, this significantly reduces the user's learning curve and improves the efficiency of device deployment. The initial calibration comprises four core steps: The first step is UWB spatial calibration, which automatically identifies the spatial positions of four sets of UWB anchor points on the TV, completes anchor point coordinate calibration and clock synchronization calibration, and constructs a precise and unified world coordinate system for the TV screen, providing a benchmark for subsequent spatial positioning calculations. The second step is simplified eye-tracking calibration, abandoning the traditional dense multi-point calibration method. Users only need to gaze at 3-5 evenly distributed standard calibration points on the screen to complete the basic calibration of eye features and gaze mapping relationships, quickly generating initial eye-tracking calibration parameters. The third step is fusion parameter learning, which records the user's initial operation data, gaze deviation patterns, and remote control holding habits to complete the adaptive learning and initialization of multi-source fusion weight parameters and matching threshold parameters. The fourth step is calibration effect verification testing, where the TV automatically executes multiple sets of positioning and interaction tests to verify spatial positioning accuracy, eye-tracking mapping accuracy, and fusion landing point stability, confirming the effectiveness and usability of the calibration parameters.

[0053] During daily operation, the device supports routine online dynamic calibration without requiring system downtime. It can correct parameter errors caused by environmental changes, user posture changes, and slight hardware drift in real time, continuously ensuring interaction accuracy. Online calibration has multiple triggering conditions. When it detects that the real-time positioning accuracy or line-of-sight matching accuracy has decreased beyond a preset accuracy threshold, when there is a significant change in the user's viewing position or holding posture, or when the system's preset periodic calibration cycle is reached, the online calibration process is automatically triggered. Online calibration includes two methods: implicit calibration and explicit calibration, adapting to different usage scenarios. Implicit calibration is completely seamless and requires no active user cooperation. During daily user interactions, it continuously collects valid line-of-sight data, remote control pointing coordinates, and matches and verifies data. The background iterates and optimizes personalized calibration parameters and integrates weight parameters in real time to dynamically compensate for system errors and parameter drift. Explicit calibration is an active and fast calibration mode. When a large accuracy deviation is detected and implicit calibration cannot correct the error, the TV pops up a window prompting the user to start fast calibration. The overall calibration time is less than 30 seconds, and parameter correction can be completed through a small number of point calibrations, quickly restoring a high-precision interactive state.

[0054] Since a television may be used by multiple users, some embodiments also include a comprehensive multi-user management mechanism. This mechanism supports independent adaptation, rapid switching, and privacy protection for multiple users, adapting to the usage scenario of multiple users sharing a television in a family, and enhancing the intelligence and user-friendliness of the device. First, it supports a quick registration function for new users. New users can complete identity entry and initial parameter settings through a simplified initial calibration upon first use, quickly generating an independent user profile. Second, it supports intelligent user switching. The system can automatically identify the current user through eye-tracking biometrics and operational behavior characteristics, automatically switching parameters. Users can also manually switch user modes, making operation flexible and convenient. Third, it has parameter inheritance and optimization capabilities. New users can match the basic parameters of existing users with similar parameters within the system, iteratively fine-tuning based on these similar parameters, significantly shortening the calibration and parameter learning cycle for new users and reducing the learning curve. Fourth, it is equipped with a comprehensive data security protection mechanism. All users' personalized calibration parameters, operational habit data, and biometric data are stored in encrypted form, preventing the leakage of user privacy data and ensuring data security and independence for multi-user use. In summary, the embodiments disclosed herein can achieve low-power operation across all scenarios through a refined power consumption optimization system, ensure long-term high-precision and stable operation of the device through lightweight initial calibration and routine online self-calibration, and adapt to the diverse needs of various user groups through a standardized multi-user management mechanism. This effectively solves the industry pain points of traditional multimodal smart TV interactive devices, such as high power consumption, cumbersome calibration, easy accuracy drift, poor multi-user adaptation, and high maintenance costs. It further improves the completeness and practicality of the entire multimodal smart TV interaction solution, significantly enhancing the long-term user experience and market competitiveness of the product. The following describes the operation process in a specific embodiment, illustrating the television interaction process provided by the embodiments of the present invention.

[0055] In one embodiment of this disclosure, the remote control is equipped with an ultra-wideband (UWB) positioning module, an inertial measurement unit (IMU) attitude acquisition module, and an infrared miniature camera. Relying on the hardware combination of UWB high-precision spatial positioning, IMU attitude calculation, and infrared eye-tracking image acquisition, and in conjunction with the TV-side multi-source fusion algorithm, adaptive mode switching, and personalized parameter adaptation mechanism, it realizes multimodal, high-precision, low-power, and self-calibrating smart TV interaction.

[0056] Figure 3 This is a flowchart illustrating a television interaction method provided in an embodiment of this disclosure. Figure 3 As shown, the process for television interaction includes: Step 301: The TV identifies the current user and determines the personalized parameters corresponding to the current user.

[0057] After the TV is turned on, the system can identify the current user's identity information through voice, avatar, etc., and retrieve the corresponding personalized eye-tracking calibration parameters, dynamic weight configuration parameters, matching time threshold, cursor sensitivity and other exclusive interaction parameters to complete the personalized initialization of the system.

[0058] Step 302: The TV obtains the current environmental information and determines whether the TV is in the converged interactive mode based on the current environmental information. If yes, proceed to step 304; otherwise, proceed to step 303.

[0059] The television collects real-time environmental information through built-in multiple sensors, covering key operating parameters such as ambient light intensity, spatial electromagnetic interference intensity, and ambient noise level. Combined with preset mode switching judgment thresholds, it adaptively identifies the current optimal interactive working mode. When sufficient ambient light, good imaging quality of the infrared eye-tracking camera, and weak spatial electromagnetic interference are detected, meeting the conditions for multimodal collaborative work, it can determine to enter the optimal fusion interactive mode and proceed to step 304. If insufficient light, strong electromagnetic interference, or other conditions that cannot meet the requirements for stable dual-modal collaborative work are detected, the television exits the fusion mode judgment.

[0060] Step 303: Determine if the TV is in pure remote control interaction mode? If yes, proceed to step 311; otherwise, proceed to step 313.

[0061] When the ambient light is dim, the user is wearing glasses or contact lenses which causes a significant decrease in eye-tracking accuracy, or the user actively selects the traditional operation mode, the system determines that it is entering a low-power, high-stability pure remote control interaction mode and proceeds to step 311. If the pure remote control mode triggering conditions are not met, the system assumes that the user does not need to point with their hand, has a need for fine selection, or has limited hand movement, and automatically determines that it is entering a pure eye-tracking interaction mode and proceeds to step 313.

[0062] Step 304: The TV acquires the current UWB spatial positioning data, current IMU inertial attitude data, and current infrared eye-tracking data uploaded by the remote control.

[0063] In the integrated interactive working mode, the remote control's UWB positioning module, IMU inertial attitude acquisition module, and infrared miniature camera are simultaneously activated and operate at high frequency, continuously acquiring multimodal raw data of spatial positioning, attitude, and eye images at a preset frame rate. The remote control performs preliminary noise reduction, timing alignment, and compression encoding on the acquired raw data, and uploads it to the TV's main control terminal in real time via a low-latency wireless communication link. The TV terminal then receives, parses, and buffers the data.

[0064] Step 305: The TV determines the current UWB spatial positioning data and current inertial attitude data, and the corresponding current three-dimensional spatial position coordinates and current three-axis attitude angular coordinates in the world coordinate system of the TV screen; and constructs the remote control spatial pointing vector based on the current three-dimensional spatial position coordinates and current three-axis attitude angular coordinates, and projects the spatial pointing vector onto the two-dimensional screen plane of the TV to obtain the coordinates of the current coarse pointing landing point of the remote control.

[0065] The television, relying on a pre-calibrated world coordinate system and four sets of synchronized UWB anchor point arrays, analyzes UWB spatial positioning data using the TDOA hyperbolic positioning algorithm to accurately calculate the real-time three-dimensional spatial coordinates of the remote control. Simultaneously, it filters and reduces noise and compensates for drift in the IMU inertial attitude data, obtaining three-axis attitude angular coordinates covering pitch, roll, and yaw angles, comprehensively representing the remote control's spatial position and handheld posture. Based on the spatial position and three-axis attitude parameters, a three-dimensional spatial pointing vector of the remote control pointing at the screen is constructed. Through spatial-plane coordinate projection transformation and normalization correction, the three-dimensional vector is mapped onto the two-dimensional plane of the television screen, ultimately outputting the current coarse pointing point coordinates of the remote control with strong stability and excellent anti-interference capabilities.

[0066] Step 306: Based on the current eye-tracking data, the TV extracts the original data of the current user's three-dimensional gaze vector and obtains the corresponding current gaze vector coordinates in the TV screen's world coordinate system; and based on the personalized eye-tracking calibration parameters matched with the current user, and using the gaze-screen mapping matrix, obtains the coordinates of the current precise eye-tracking positioning point corresponding to the current gaze vector coordinates.

[0067] The television performs grayscale conversion, filtering and noise reduction, image enhancement, and eye region cropping preprocessing on the received infrared eye-tracking image data. It identifies the pupil contour and center position using an ellipse fitting algorithm, tracks corneal reflection feature points using optical flow, and calculates the user's three-dimensional gaze vector based on the pupil-corneal reflection vector principle. Through coordinate system transformation, the camera's local gaze vector is corrected to the standard gaze vector coordinates in the television screen's world coordinate system. Personalized eye-tracking calibration parameters specific to the current user are retrieved to compensate for individual errors. Matrix transformation operations are then performed using a pre-trained gaze-screen mapping matrix to ultimately obtain the sub-pixel-level high-precision coordinates of the current eye-tracking positioning point.

[0068] Step 307: The TV determines the data source signal-to-noise ratio and data confidence level corresponding to the current remote control coarse pointing point coordinates and the current eye-tracking precise positioning point coordinates, respectively; based on the current ambient light intensity, data source signal-to-noise ratio, and data confidence level, the TV determines the weight values ​​corresponding to the current remote control coarse pointing point coordinates and the current eye-tracking precise positioning point coordinates, respectively.

[0069] The television system performs real-time quantitative evaluation of the quality status of two positioning data sources, calculating the real-time signal-to-noise ratio (SNR) and data confidence of coarse pointing data and fine eye-tracking data respectively. This quantitatively characterizes the effective signal ratio, stability, and reliability of the two types of data. Combined with real-time ambient light intensity, electromagnetic interference levels, and other environmental conditions, the system adaptively performs dynamic weight allocation: when lighting is excellent, eye-tracking data has a high SNR and high confidence, the weight of fine eye-tracking data is increased; when lighting is poor, electromagnetic interference is strong, and UWB data fluctuations are small, the weight of UWB+IMU coarse positioning data is increased, achieving complementary advantages and dynamic adaptation between the two data sources.

[0070] Step 308: The TV performs time synchronization and coordinate benchmark unification processing on the current remote control coarse pointing point coordinates and the current eye-tracking precise positioning point coordinates, and performs dynamic weight allocation and fusion according to the determined weight values ​​to obtain fused position information.

[0071] Step 309: Based on the fused location information, the TV constructs a current system state vector containing the remote control's spatial position, attitude angle, and screen landing point coordinates. It then uses an extended Kalman filter for state prediction and observation correction, and iteratively optimizes to obtain the current screen fused pointing landing point coordinates.

[0072] Based on the initial fused position information, the television integrates the remote control's real-time spatial position parameters, three-axis attitude angle parameters, and the initial fused landing point parameters of the screen to construct a complete system state vector, comprehensively representing the current operating state of the interactive system. An extended Kalman filter algorithm is used to iteratively optimize the system state. Through a closed-loop mechanism of state prediction, observation update, and error correction, it continuously compensates for system steady-state drift, random noise, and environmental interference errors, constantly optimizing the accuracy and stability of the landing point coordinates. Finally, iterative convergence yields high-precision, low-drift, and highly interference-resistant current screen fused pointing landing point coordinates.

[0073] Step 310: If the current eye-tracking precise positioning point coordinates match the current screen fusion pointing point coordinates, and the matching time is longer than the matching time threshold, the TV will perform the corresponding TV interaction based on the current screen fusion pointing point coordinates, and this process will end.

[0074] The TV compares the coordinate deviation between the eye-tracking precise positioning point and the screen fusion pointing point in real time. When the deviation is less than a preset matching threshold, it determines that the user's gaze and the remote control's pointing point are highly aligned and focused on the same target area, entering a valid matching timer state. The system continuously tracks the duration of continuous matching. When the duration exceeds the user's personalized matching time threshold, it filters out false triggering behaviors such as slight eye movements and unintentional glances, accurately determining the user's valid interaction intent. Based on the high-precision fusion point coordinates, the TV automatically executes corresponding TV interaction operations such as menu selection, function activation, parameter confirmation, and interface switching, completing a full intelligent interaction process in fusion mode, at which point the process ends.

[0075] Step 311: The TV obtains the current UWB spatial positioning data and the current IMU inertial attitude data uploaded by the remote control, and obtains the corresponding coordinates of the current coarse pointing point of the remote control.

[0076] In pure remote control interaction mode, the infrared eye-tracking camera can be turned off or put into a low-power sleep state, leaving only the UWB positioning module and IMU inertial attitude module to operate normally, reducing the overall power consumption of the device. The TV receives UWB spatial positioning data and IMU inertial attitude data uploaded by the remote control in real time. Through the TDOA hyperbolic positioning algorithm and attitude calculation logic, the TV calculates the three-dimensional spatial coordinates and three-axis attitude angles of the remote control, constructs a spatial pointing vector, and projects it onto the two-dimensional plane of the screen. It stably outputs the coordinates of the current coarse pointing point of the remote control, ensuring the continuous availability of interactive functions in extreme scenarios such as dim lighting and complex electromagnetic interference.

[0077] Step 312: The TV performs the corresponding TV interaction based on the coordinates of the current coarse pointer on the remote control. This process ends.

[0078] In pure remote control mode, the coarse pointer coordinates obtained from UWB+IMU calculation serve as the sole interaction reference. The cursor follows the coarse pointer's movement in real time. Users can trigger interactions using traditional methods such as physical buttons on the remote control or long-press confirmation. The TV executes corresponding interaction commands such as selection, switching, and confirmation based on the real-time coarse pointer coordinates. This mode boasts the lowest power consumption, extremely high stability, and no light dependence, perfectly compatible with traditional operating habits. The process ends upon completion of the interaction.

[0079] Step 313: The TV obtains the current infrared eye-tracking data uploaded by the remote control and gets the corresponding current precise eye-tracking positioning coordinates.

[0080] In pure eye-tracking interaction mode, the UWB positioning module can be turned off and put into a low-power sleep state, with only the infrared miniature camera continuously and frequently collecting user eye image data and uploading it to the TV. The TV relies on a refined eye-tracking algorithm to complete eye image preprocessing, feature detection, gaze vector calculation, coordinate calibration, and screen mapping. Combined with the user's personalized calibration parameters, it obtains high-precision, sub-pixel-level coordinates of the current eye movement positioning point, eliminating the need for remote control pointing and adapting to special scenarios such as limited hand movement and the selection of fine, small targets.

[0081] Step 314: The TV accurately locates the landing point coordinates based on the current eye movement and performs the corresponding TV interaction, thus ending this process.

[0082] In pure eye-tracking mode, the eye movement precisely locates the coordinates of the cursor as the interaction reference. The cursor is directly and accurately positioned at the user's gaze location, and the user's interaction intent is automatically determined based on the gaze duration threshold. Selection and confirmation can be completed autonomously without hand operation. This mode offers natural operation and high positioning accuracy, meeting the needs of sophisticated interface operation. The TV executes the corresponding interaction command based on the real-time eye movement precisely located coordinates, and the process ends after the intelligent interaction is completed.

[0083] As can be seen, this embodiment constructs a high-precision, robust, low-power, and all-scenario-adaptive smart TV interaction system through a complete technical chain of multimodal sensor collaborative acquisition, independent dual-landing point calculation, environmentally adaptive dynamic weight fusion, and Kalman filter iterative optimization, combined with a three-level branched interactive mode adaptive switching mechanism. This solution innovatively integrates the stability advantages of UWB and IMU coarse positioning with the high-precision advantages of infrared eye-tracking fine positioning. Through a multi-dimensional dynamic weighting fusion strategy considering ambient light, electromagnetic interference, and data confidence, it effectively overcomes the shortcomings of single positioning methods, such as susceptibility to environmental interference, fixed accuracy, and insufficient stability, significantly improving the accuracy and smoothness of screen landing point calculation. Simultaneously, this embodiment, through user identification and personalized parameter adaptation mechanisms, differentiates calibration parameters, fusion weights, matching thresholds, and cursor sensitivity based on different users' eye characteristics and operating habits, solving the problems of poor experience, high false trigger rate, and low adaptability caused by adapting universal parameters to all users in traditional interaction solutions. By intelligently switching between three interaction modes—interactive, pure remote control, and pure eye-tracking—it can adaptively adapt to various complex scenarios and user needs, including strong light, low light, electromagnetic interference, hand limitations, fine-tuning, and traditional controls. It balances high-precision intelligent interaction in daily use, stable interaction in extreme environments, and seamless interaction in special scenarios. Furthermore, with module-level sleep management, dynamic frequency adjustment power optimization strategies, and an online self-calibration maintenance mechanism, it effectively reduces overall power consumption while ensuring real-time and accurate interaction. It also enables parameter self-correction over long-term use, avoiding accuracy degradation caused by hardware drift and environmental changes. The overall interaction logic is clear, fault-tolerant, and comprehensively covers scenarios, completely solving the industry pain points of traditional TV interaction—cumbersome operation, fixed modes, limited precision, weak environmental adaptability, and poor user compatibility—significantly improving the naturalness, accuracy, stability, and intelligence level of smart TV human-computer interaction.

[0084] Based on the above process for television interaction, a device for television interaction can be constructed.

[0085] Figure 4 This is a schematic diagram of a structure for a television interactive device provided in an embodiment of this disclosure. Figure 4 As shown, the television interactive device includes: a positioning acquisition module 410, a fusion positioning module 420, and a first interactive module 430.

[0086] The positioning module 410 is configured to acquire the current spatial positioning data, current inertial attitude data, and current eye-tracking data uploaded by the remote control when the TV is in the converged interactive mode, and obtain the corresponding coordinates of the current coarse pointing point of the remote control and the current precise eye-tracking positioning point, respectively. The spatial positioning data includes one or more of the following: UWB positioning data, Bluetooth AoA positioning data, Wi-Fi positioning data, and infrared positioning data.

[0087] The fusion positioning module 420 is configured to obtain the current screen fusion pointing landing point coordinates based on the current remote control coarse pointing landing point coordinates and the current eye-tracking precise positioning landing point coordinates, using a multi-source sensor fusion filtering algorithm for iterative estimation. The multi-source sensor fusion filtering algorithm includes one or more of the following: extended Kalman filter algorithm, particle filter algorithm, neural network fusion algorithm, and fuzzy logic fusion algorithm.

[0088] The compensation control module 430 is configured to perform corresponding TV interaction based on the current screen fusion pointing point coordinates when it is determined that the current eye-tracking precise positioning point coordinates match the current screen fusion pointing point coordinates.

[0089] In some embodiments, the positioning module 410 includes: The coordinate determination unit is configured to determine the current spatial positioning data and the current inertial attitude data, and the corresponding current three-dimensional spatial position coordinates and current three-axis attitude angular coordinates in the world coordinate system of the television screen.

[0090] The vector positioning unit is configured to construct a spatial pointing vector for the remote control based on the current three-dimensional spatial position coordinates and the current three-axis attitude angular coordinates, and project the spatial pointing vector onto the two-dimensional screen plane of the TV to obtain the coordinates of the current coarse pointing point of the remote control.

[0091] In some embodiments, the positioning module 410 includes: The extraction and transformation unit is configured to extract the raw data of the current user's three-dimensional gaze vector based on the current eye tracking data, and obtain the corresponding current gaze vector coordinates in the world coordinate system of the TV screen.

[0092] The matrix positioning unit is configured to obtain the coordinates of the current eye movement precise positioning point corresponding to the current human eye's gaze vector coordinates based on the gaze screen mapping matrix, according to the personalized eye movement calibration parameters matched with the current user.

[0093] In some embodiments, the fusion positioning module 420 includes: The alignment weight unit is configured to perform time synchronization alignment and coordinate benchmark unification processing on the current remote control coarse pointing landing point coordinates and the current eye-tracking precise positioning landing point coordinates, and then perform dynamic weight allocation and fusion to obtain fused position information.

[0094] An iterative unit is constructed and configured to build a current system state vector containing the remote controller's spatial position, attitude angle, and screen landing point coordinates based on the fused position information. The state is predicted and observed is corrected by extended Kalman filtering, and the current screen fusion pointing landing point coordinates are obtained through iterative optimization.

[0095] In some embodiments, the alignment weighting unit is specifically configured to determine the data source signal-to-noise ratio and data confidence level corresponding to the current remote control coarse pointing landing point coordinates and the current eye-tracking precise positioning landing point coordinates, respectively; based on the current ambient light intensity, data source signal-to-noise ratio, and data confidence level, the weight values ​​corresponding to the current remote control coarse pointing landing point coordinates and the current eye-tracking precise positioning landing point coordinates are determined respectively, thereby completing the dual-landing point adaptive weighted fusion.

[0096] In some embodiments, it also includes: The identification and determination module is configured to identify the current user and determine the personalized parameters corresponding to the current user. The personalized parameters include one or more of the following: matching time threshold, personalized eye-tracking calibration parameters, weight value, and cursor sensitivity parameters.

[0097] In some embodiments, it also includes: The mode determination module is configured to determine the current interaction mode of the TV when preset environmental conditions are met or when a user operation command is received.

[0098] The second interaction module is configured to perform corresponding TV interactions based on the coordinates of the current coarse pointer of the remote control when the TV is in pure remote control interaction mode.

[0099] The third interaction module is configured to perform corresponding TV interactions based on the precise coordinates of the current eye movement location when the TV is in pure eye-tracking interaction mode.

[0100] The following example illustrates the television interaction process of the device for television interaction provided in the embodiments of the present invention.

[0101] In one embodiment of this disclosure, the device for television interaction can be applied to a television. The remote control is equipped with an ultra-wideband (UWB) positioning module, an IMU inertial attitude acquisition module, and an infrared miniature camera. Relying on the hardware combination of UWB high-precision spatial positioning, IMU attitude calculation, and infrared eye-tracking image acquisition, and in conjunction with a multi-source fusion algorithm, adaptive mode switching, and personalized parameter adaptation mechanism on the television side, multi-modal, high-precision, low-power, and self-calibrating smart television interaction is achieved. Figure 5 As shown, the television interactive device includes: an acquisition and positioning module 410, a fusion positioning module 420, a first interaction module 430, an identification and determination module 440, a pattern determination module 450, a second interaction module 460, and a third interaction module 470. The acquisition and positioning module 410 includes: a coordinate determination unit 411, a vector positioning unit 412, an extraction and transformation unit 413, and a matrix determination unit 414. The fusion positioning module 420 includes: an alignment weight unit 421 and a construction iteration unit 422.

[0102] After the TV is turned on, the voice module configured on the TV acquires the current voice information. Based on this, the recognition and determination module 440 can identify the current user and determine the personalized parameters corresponding to that user. These personalized parameters include: matching time threshold, personalized eye-tracking calibration parameters, weight values, and cursor sensitivity parameters. Simultaneously, the TV uses built-in multiple sensors to collect real-time environmental information, covering key operating parameters such as ambient light intensity, spatial electromagnetic interference intensity, and ambient noise level. Combined with preset modes, the mode determination module 450 can determine the corresponding operating mode of the TV. Specifically, when sufficient ambient light, good imaging quality of the infrared eye-tracking camera, and weak spatial electromagnetic interference are detected, meeting the conditions for multimodal collaborative operation, the mode determination module 450 can determine that the TV is in the fused interactive mode; if insufficient light, strong electromagnetic interference, or other conditions that cannot meet the requirements for stable dual-modal collaborative operation are detected, and the ambient light is dim, the mode determination module 450 can determine that the TV is in the pure remote control interactive mode; if strong electromagnetic interference cannot meet the requirements for stable dual-modal collaborative operation, and sufficient light is detected, the mode determination module 450 can determine that the TV is in the pure eye-tracking interactive mode.

[0103] When the TV is in the converged interactive mode, the positioning module 410 acquires the current UWB spatial positioning data, current IMU inertial attitude data, and current infrared eye-tracking data uploaded by the remote control. Specifically, the coordinate determination unit 411 determines the current UWB spatial positioning data and current inertial attitude data, as well as the corresponding current three-dimensional spatial position coordinates and current three-axis attitude angular coordinates in the TV screen's world coordinate system. The vector positioning unit 412 constructs the remote control's spatial pointing vector based on the current three-dimensional spatial position coordinates and current three-axis attitude angular coordinates, and projects the spatial pointing vector onto the TV's two-dimensional screen plane to obtain the current remote control's coarse pointing landing point coordinates. Furthermore, the extraction and conversion unit 413 extracts the original data of the current user's three-dimensional gaze vector based on the current eye-tracking data, and obtains the corresponding current human eye gaze vector coordinates in the TV screen's world coordinate system. The matrix positioning unit 414, based on personalized eye-tracking calibration parameters matched with the current user and a gaze-screen mapping matrix, obtains the current precise eye-tracking positioning landing point coordinates corresponding to the current human eye gaze vector coordinates.

[0104] Therefore, the alignment weight unit 421 in the fusion positioning module 420 can determine the data source signal-to-noise ratio and data confidence level corresponding to the current remote control coarse pointing landing point coordinates and the current eye-tracking precise positioning landing point coordinates, respectively. Based on the current ambient light intensity, data source signal-to-noise ratio, and data confidence level, it determines the weight values ​​corresponding to the current remote control coarse pointing landing point coordinates and the current eye-tracking precise positioning landing point coordinates, and performs time synchronization alignment and coordinate reference unification processing on the current remote control coarse pointing landing point coordinates and the current eye-tracking precise positioning landing point coordinates. Based on the determined weight values, it performs dynamic weight allocation fusion to obtain fused position information. Furthermore, the construction iteration unit 422 constructs a current system state vector containing the remote control's spatial position, attitude angle, and screen landing point coordinates based on the fused position information. It performs state prediction and observation correction through extended Kalman filtering, and iteratively optimizes to obtain the current screen fused pointing landing point coordinates. Therefore, when it is determined that the current eye-tracking precise positioning landing point coordinates and the current screen fused pointing landing point coordinates match, if the continuous matching time exceeds a matching time threshold, the first interaction module 430 performs the corresponding TV interaction based on the current screen fused pointing landing point coordinates.

[0105] When the TV is in pure remote control interaction mode, it only needs to obtain the current UWB spatial positioning data and the current IMU inertial attitude data uploaded by the remote control, and obtain the corresponding current coarse pointing landing point coordinates of the remote control. In this way, the second interaction module 460 can perform corresponding TV interaction based on the current coarse pointing landing point coordinates of the remote control. When the TV is in pure eye-tracking interaction mode, it can only obtain the current infrared eye-tracking data uploaded by the remote control and obtain the corresponding current eye-tracking precise positioning point coordinates. Then, the third interaction module performs the corresponding TV interaction based on the current eye-tracking precise positioning point coordinates.

[0106] As can be seen, in this embodiment, the device disclosed for television interaction, through a modular and unitized hardware architecture and a hierarchical functional logic design, relies on the collaborative cooperation of the identification and determination module, the mode determination module, the multi-type data acquisition and positioning unit, and the fusion positioning module to fully realize the full-link device functions of multimodal data acquisition, personalized parameter adaptation, adaptive mode switching, accurate dual-landing point calculation, dynamic weight fusion, and intelligent interaction execution. This device achieves stable coarse positioning output through the coordinate determination unit and the vector positioning unit, and achieves high-precision eye-tracking line-of-sight landing point calculation through the extraction and transformation unit and the matrix determination unit. It completes adaptive weighted fusion and filtering iterative optimization of multi-source data by relying on the alignment weight unit and the construction iteration unit, effectively integrating the dual advantages of remote control pointing positioning stability and eye-tracking positioning high precision, overcoming the shortcomings of single interactive devices such as limited positioning accuracy, weak anti-interference ability, and poor environmental adaptability. Meanwhile, this device can adaptively switch between three interactive modes—fusion interaction, pure remote control, and pure eye-tracking—based on ambient light, electromagnetic interference, and other operating conditions. It can adapt to various complex usage scenarios, such as dim lighting, complex electromagnetic interference, delicate operation, and hand limitations, solving the problems of traditional TV interactive devices' single function, fixed modes, and poor scenario adaptability. Furthermore, the device relies on a recognition module to complete multi-user identity recognition and personalized parameter matching. It can adaptively calibrate interaction parameters according to different users' eye characteristics and operating habits, significantly reducing the probability of accidental interaction triggers and improving human-computer adaptation and naturalness of use. The overall device architecture is clear, with low functional coupling and strong scalability. While ensuring interaction accuracy and real-time response, it also features low-power operation, self-calibration and maintenance, and multi-user adaptation capabilities, effectively improving the intelligence, operational stability, and scenario versatility of smart TV interactive devices, and greatly optimizing the user's smart TV control experience.

[0107] This disclosure provides a device 600 for television interaction, the structure of which is as follows: Figure 6 As shown, it includes: The processor 1000 and memory 1001 may further include a communication interface 1002 and a bus 1003. The processor 1000, communication interface 1002, and memory 1001 can communicate with each other via the bus 1003. The communication interface 1002 can be used for information transmission. The processor 1000 can call logical instructions stored in the memory 1001 to execute the television interaction method described in the above embodiments.

[0108] Furthermore, the logic instructions in the aforementioned memory 1001 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0109] The memory 1001, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 1000 executes functional applications and data processing by running the program instructions / modules stored in the memory 1001, that is, it implements the method for television interaction in the above method embodiments.

[0110] The memory 1001 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 1001 may include high-speed random access memory and may also include non-volatile memory.

[0111] This disclosure provides a television interaction device, including: a processor and a memory storing program instructions, wherein the processor is configured to execute a television interaction method when executing the program instructions.

[0112] This disclosure provides a system for television interaction, such as... Figure 1 As shown, it includes a television and a remote control, wherein the television can be used as follows: Figure 7 As shown, the device includes a device body 700, in which the aforementioned device 400 (600) for television interaction is installed. The installation relationship described herein is not limited to placement within the product, but also includes installation connections with other components of the product, including but not limited to physical connections, electrical connections, or signal transmission connections. Those skilled in the art will understand that the device 400 (600) for television interaction can be adapted to suitable corresponding device bodies to achieve other feasible embodiments.

[0113] This disclosure provides a storage medium storing program instructions that, when executed, perform the method for television interaction as described above.

[0114] This disclosure provides a computer program product, which includes a computer program stored on a storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the above-described television interaction method.

[0115] The aforementioned storage medium can be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0116] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0117] The foregoing description and accompanying drawings fully illustrate embodiments of the present disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included or replace parts and features of other embodiments. The scope of the embodiments of this disclosure includes the entire scope of the claims and all available equivalents of the claims. While the terms “first,” “second,” etc., may be used in this application to describe elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first element may be called a second element without changing the meaning of the description, and similarly, a second element may be called a first element, provided that all occurrences of “first element” are consistently renamed and all occurrences of “second element” are consistently renamed. First and second elements are both elements, but may not be the same element. Moreover, the terminology used in this application is only for describing embodiments and is not intended to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” are intended to also include the plural forms. Similarly, the term “and / or” as used herein means including one or more of the associated listed any and all possible combinations. Additionally, when used herein, the terms “comprise” and its variations “comprises” and / or “comprising” refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase “comprising an…” does not exclude the presence of additional identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0118] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0119] The methods and products disclosed in the embodiments herein (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0120] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A method for television interaction, characterized in that, include: When the TV is in the converged interactive mode, the system acquires the current spatial positioning data, current inertial attitude data, and current eye-tracking data uploaded by the remote control, and obtains the corresponding coordinates of the current coarse pointing point of the remote control and the coordinates of the current precise eye-tracking point. The spatial positioning data includes one or more of the following: UWB positioning data, Bluetooth AoA positioning data, Wi-Fi positioning data, and infrared positioning data. Based on the current coarse pointing coordinates of the remote control and the current precise eye-tracking positioning coordinates, the current screen fusion pointing coordinates are obtained through iterative estimation using a multi-source sensor fusion filtering algorithm. The multi-source sensor fusion filtering algorithm includes one or more of the following: extended Kalman filter algorithm, particle filter algorithm, neural network fusion algorithm, and fuzzy logic fusion algorithm. If the current eye-tracking precise positioning point coordinates match the current screen fusion pointing point coordinates, and the matching time exceeds the matching time threshold, then the corresponding TV interaction is performed based on the current screen fusion pointing point coordinates.

2. The method according to claim 1, characterized in that, The obtained coordinates of the current coarse pointing point of the remote control include: Determine the current spatial positioning data and current inertial attitude data, and the corresponding current three-dimensional spatial position coordinates and current three-axis attitude angular coordinates in the world coordinate system of the TV screen; Based on the current three-dimensional spatial position coordinates and the current three-axis attitude angular coordinates, a spatial pointing vector of the remote control is constructed. The spatial pointing vector is then projected onto the two-dimensional screen plane of the TV to obtain the coordinates of the current coarse pointing point of the remote control.

3. The method according to claim 1, characterized in that, The obtained coordinates of the current precise eye-tracking positioning point include: Based on the current eye-tracking data, extract the raw data of the current user's three-dimensional gaze vector and obtain the corresponding current gaze vector coordinates in the world coordinate system of the TV screen; Based on the personalized eye-tracking calibration parameters matched with the current user, and using the gaze-screen mapping matrix, the coordinates of the current precise eye-tracking positioning point corresponding to the current human eye gaze vector coordinates are obtained.

4. The method according to claim 1, characterized in that, The process of obtaining the coordinates of the current screen fusion target landing point includes: The coordinates of the current coarse pointing point of the remote control and the coordinates of the current precise eye-tracking point are synchronized in time and the coordinate reference is unified. Dynamic weight allocation is then performed to fuse the coordinates to obtain the fused position information. Based on the fused location information, a current system state vector is constructed, which includes the remote controller's spatial position, attitude angle, and screen landing point coordinates. The state is predicted and the observation is corrected by extended Kalman filtering. The current screen fused pointing landing point coordinates are obtained through iterative optimization.

5. The method according to claim 4, characterized in that, The dynamic weight allocation and fusion process includes: Determine the signal-to-noise ratio and data confidence level of the data source corresponding to the current coarse pointing coordinates of the remote control and the current precise eye-tracking positioning coordinates, respectively; Based on the current ambient light intensity, data source signal-to-noise ratio, and data confidence level, the weight values ​​corresponding to the current remote control coarse pointing landing point coordinates and the current eye-tracking precise positioning landing point coordinates are determined respectively, and the dual landing point adaptive weighted fusion is completed.

6. The method according to claim 1, characterized in that, Also includes: The system identifies the current user and determines the personalized parameters corresponding to the current user. The personalized parameters include one or more of the following: matching time threshold, personalized eye-tracking calibration parameters, weight values, and cursor sensitivity parameters.

7. The method according to any one of claims 1-6, characterized in that, Also includes: Under the condition that the preset environmental conditions are met or the user operation command is received, determine the current interaction mode of the TV; When the TV is in pure remote control interaction mode, the corresponding TV interaction is performed according to the coordinates of the current coarse pointer of the remote control. When the TV is in pure eye-tracking interaction mode, the corresponding TV interaction is performed based on the precise coordinates of the current eye-tracking location.

8. A device for television interaction, the device comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to perform the television method as described in any one of claims 1 to 7 when executing the program instructions.

9. A system for television interaction, characterized in that, include: Remote control and television; among them, The remote control is configured to collect current UWB positioning data, current inertial attitude data, and current eye-tracking data through a configured spatial positioning acquisition module, an inertial attitude acquisition module, and an eye-tracking acquisition module, and then upload them to the television. The spatial positioning acquisition module includes one or more of a UWB module, a Bluetooth positioning module, a Wi-Fi positioning module, and an infrared positioning module; the eye-tracking acquisition module includes one or more of an infrared miniature camera, a visible light RGB camera, a structured light camera, and an event camera. The television is configured to, when in fusion interaction mode, acquire current spatial positioning data, current inertial attitude data, and current eye-tracking data uploaded by the remote control, and obtain the corresponding current coarse remote control pointing coordinates and current precise eye-tracking pointing coordinates; based on the current coarse remote control pointing coordinates and current precise eye-tracking pointing coordinates, iteratively predict the current fusion pointing coordinates on the screen using a multi-source sensor fusion filtering algorithm; if the current precise eye-tracking pointing coordinates match the current fusion pointing coordinates on the screen, and if the matching time exceeds a matching time threshold, perform corresponding television interaction based on the current fusion pointing coordinates on the screen. The multi-source sensor fusion filtering algorithm includes one or more of the following: extended Kalman filter algorithm, particle filter algorithm, neural network fusion algorithm, and fuzzy logic fusion algorithm.

10. A storage medium storing program instructions, characterized in that, When the program instructions are executed, they perform the method for television interaction as described in any one of claims 1 to 7.

Images