Display system and display method

Abstract

The invention provides a display system and a display method. The system comprises a display terminal, an infrared marking module, a remote control module and a processing module, the remote control module comprises an inertia sensing unit and an event camera; the event camera is used for sensing infrared light spots emitted by the infrared light-emitting device and generating position information and confidence degree information of the infrared light-emitting device based on the infrared light spots; the inertia sensing unit is used for sensing attitude data of the remote control module; and the processing module is used for determining yaw information and translation information of the remote control module relative to the display terminal based on the position information, the confidence coefficient information and the attitude data, and determining coordinates of an intersection point of the optical axis and a display plane of the display terminal according to the yaw information, the translation information and the attitude data. Therefore, the coordinate of the intersection point of the optical axis and the display plane is calculated by combining the infrared light point generation position and confidence information sensed by the event camera and fusing the attitude data of the inertial sensing unit, so that the cursor is stably controlled, and the effect of improving the stability and precision of cursor control is achieved.

Description

Technical Field

[0001] This application relates to the field of display technology, specifically to a display system and display method. Background Technology

[0002] In the field of human-computer interaction and sensor fusion, screen pointing technology, as a bridge connecting users and display terminals, is widely used in scenarios such as LCD displays. Traditional display control schemes based on human-computer interaction and sensor fusion mainly rely on inertial measurement units for attitude extrapolation and filtering fusion, or on solving the screen attitude and coordinate mapping relationship through homography transformation based on multiple reference points arranged outside the screen. However, in practical applications, for large-screen TV scenarios, the physical spacing between reference points outside the screen is often too large, causing the field of view of the frame camera or event camera to be unable to cover all reference points simultaneously. Especially when the remote control and the screen form a large yaw angle or the operating distance is close, some reference points are very likely to exceed the camera's field of view boundary, resulting in missing reference points. When only one or two reference points are visible, the system cannot stably solve the screen coordinate mapping, the interaction process is frequently interrupted, and the user experience is seriously affected. Summary of the Invention

[0003] This application provides a display system and display method that improves the stability and accuracy of cursor control, reduces operation interruptions and drift, and enhances the user experience.

[0004] In a first aspect, the display system provided in this application includes: a display terminal, an infrared marking module, a remote control module, and a processing module; the infrared marking module includes at least two infrared light-emitting devices arranged based on the position of the display terminal; the remote control module includes an inertial sensing unit and an event camera, the optical axis of the event camera being oriented towards the display terminal; The event camera is used to sense the infrared light spots emitted by the infrared light-emitting device, and to generate the position information and confidence information of the infrared light-emitting device based on the infrared light spots; The inertial sensing unit is used to sense the attitude data of the remote control module; The processing module is configured to determine the yaw and translation information of the remote control module relative to the display terminal based on the position information, the confidence information, and the attitude data; determine the intersection coordinates of the optical axis and the display plane of the display terminal based on the yaw information, the translation information, and the attitude data; and control the cursor of the user interface of the display terminal based on the intersection coordinates.

[0005] Secondly, the display method provided in this application embodiment is applied to a display system, and the method includes: The infrared light spot emitted by the infrared light-emitting device is sensed, and the position information and confidence information of the infrared light-emitting device are generated based on the infrared light spot; Sensing attitude data from the remote control module; Based on the location information, the confidence information, and the attitude data, the yaw and translation information of the remote control module relative to the display terminal are determined. The intersection coordinates of the optical axis of the event camera and the display plane of the display terminal are determined according to the yaw information, the translation information, and the attitude data. The cursor of the user interface of the display terminal is controlled according to the intersection coordinates.

[0006] In summary, the display system and method provided in this application combine the position and confidence information of the infrared light point sensed by the event camera with the attitude data of the inertial sensing unit to calculate the coordinates of the intersection point of the optical axis and the display plane, thereby stabilizing the cursor control. This has the advantages of improving the stability and accuracy of cursor control, reducing operation interruption and drift, and enhancing the user experience. Attached Figure Description

[0007] The present invention will be further described below with reference to the accompanying drawings. It should be noted that the accompanying drawings described below are merely for illustrating some embodiments of the present invention. Those skilled in the art can obtain other drawings based on the above drawings without any creative effort.

[0008] Figure 1 A schematic diagram of a display system provided for an embodiment of this application.

[0009] Figure 2 This is an exemplary schematic diagram of an infrared light-emitting device in an embodiment of this application.

[0010] Figure 3 This is a schematic diagram of the working process of the infrared transistor in the embodiments of this application.

[0011] Figure 4 This is a schematic diagram illustrating the workflow of the event camera and inertial sensing unit in the embodiments of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0013] In this invention, the terms "first," "second," etc., are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to the aforementioned process, method, product, or apparatus.

[0014] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily imply that all embodiments are the same, nor are they independent or alternative embodiments mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0015] This application provides a display system, which includes, but is not limited to, the following embodiments and combinations thereof.

[0016] In one embodiment, Figure 1 A schematic diagram of a display system provided for an embodiment of this application, as shown below. Figure 1 As shown, the display system 100 includes a display terminal 101, an infrared marking module 102, a remote control module 103, and a processing module 104; the infrared marking module 102 includes at least two infrared light-emitting devices arranged based on the position of the display terminal 101; the remote control module 103 includes an inertial sensing unit and an event camera, with the optical axis of the event camera facing the display terminal 101. An event camera is used to sense infrared light spots emitted by infrared light-emitting devices and to generate position and confidence information of the infrared light-emitting devices based on these light spots.

[0017] An inertial sensing unit is used to sense the attitude data of the remote control module.

[0018] The processing module 104 is used to determine the yaw and translation information of the remote control module relative to the display terminal 101 based on the position information, confidence information and attitude data, determine the intersection coordinates of the optical axis and the display plane of the display terminal 101 according to the yaw information, translation information and attitude data, and control the cursor of the user interface of the display terminal 101 according to the intersection coordinates.

[0019] The display terminal 101 can be any type of display device, such as a liquid crystal display, an OLED display, or a projector. The infrared marking module 102 can consist of multiple independent infrared light-emitting devices, which can be fixed to the frame of the display terminal 101, such as at the four corners, or evenly arranged along the edge of the display terminal 101. The infrared light-emitting devices can emit light continuously or flash according to a preset pattern. The remote control module 103 can be a handheld device. The remote control module 103 includes an inertial sensing unit and an event camera, with the optical axis of the event camera facing the display terminal 101. The inertial sensing unit can be a combination of a three-axis gyroscope and a three-axis accelerometer, used to provide the angular velocity and linear acceleration of the remote control module 103, and then estimate the real-time attitude of the remote control module 103 by integrating the angular velocity and acceleration data. The event camera can be an asynchronous event vision sensor, with its lens designed to have a certain field of view to cover the entire area of ​​the display terminal 101. The optical axis of the event camera can be calibrated to approximately point towards the center area of ​​the display terminal.

[0020] Processing module 104 determines the yaw and translation information of remote control module 103 relative to display terminal 101 based on position information, confidence information, and attitude data. Processing module 104 can establish a geometric model, combining the position of the infrared light spot sensed by the event camera in the camera coordinate system with the attitude data of remote control module 103, and estimate the three-dimensional position and attitude of remote control module 103 relative to display terminal 101, including yaw and translation, through triangulation. Confidence information can be used to weight different light spot data to improve the robustness of the estimation. Further, processing module 104 determines the intersection coordinates of the optical axis and the display plane of display terminal 101 based on yaw information, translation information, and attitude data. With the complete attitude and position (yaw and translation) of remote control module 103 relative to display terminal 101 determined, processing module 104 can calculate the projection point of the optical axis of the event camera onto the display plane of display terminal 101 by transforming the optical axis vector in the camera coordinate system to the display terminal 101 coordinate system, and then calculating the intersection point of the vector and the display plane equation. Thus, the processing module 103 controls the cursor of the user interface on the display terminal 101 based on the intersection point coordinates. The obtained intersection point coordinates can be directly mapped to the user interface coordinate system of the display terminal 101 to control the movement of the cursor. For example, the physical size of the display plane can be mapped to the screen pixel resolution, thereby converting the physical intersection point coordinates into screen pixel coordinates, and thus driving the cursor.

[0021] This application utilizes an event camera to sense the position and confidence information of an infrared light spot, and combines this with the attitude data of the remote control module provided by an inertial sensing unit. This enables precise and stable determination of the yaw and translation information of the remote control module relative to the display terminal in large-size display terminal scenarios. Consequently, the coordinates of the intersection point between the optical axis and the display plane can be accurately calculated, effectively overcoming problems such as missing light spots, limited field of view, and inertial drift in traditional solutions. This achieves high real-time, high-precision cursor pointing control, enhancing the user's interactive experience on large-size screens.

[0022] In some embodiments of this application, the accuracy and reliability of position information and attitude data may dynamically change due to factors such as changes in ambient lighting, the motion state of the remote control module, or the characteristics of the sensors themselves. Simply fusing the data may result in insufficient accuracy and stability in the calculation of yaw and translation information, thereby affecting the smoothness and accuracy of cursor control.

[0023] Based on this, in one embodiment, the processing module 104 is further configured to: adjust the weights of position information and attitude data using a preset weight fusion strategy, and update yaw information and translation information.

[0024] The preset weight fusion strategy can be a predefined method or algorithm for intelligently combining multiple data sources. It can assign different levels of importance (i.e., weights) to each data source based on its characteristics or real-time performance. The weights can be fixed values ​​or dynamically adjusted according to preset rules.

[0025] The processing module 104 optimizes the fusion effect by adjusting the weights of position information and attitude data. When calculating yaw and translation information, the two types of data are not simply treated equally; instead, they are assigned different influences based on their respective reliability, accuracy, or contribution to the current state. For example, when the infrared light spots sensed by the event camera are clear, numerous, and have high confidence, the generated position information can be given a higher weight; conversely, when the infrared light spots are blurry or have low confidence, the weight of the position information can be reduced, and the weight of the attitude data sensed by the inertial sensing unit can be increased accordingly. Furthermore, when the remote control module 103 moves rapidly, the instantaneous attitude data from the inertial sensing unit may be more accurate, and the weight of the attitude data can be appropriately increased in this case. The weight adjustment can be statically preset or dynamically adjusted based on real-time sensor performance indicators (such as the confidence information of the event camera and the noise level of the inertial sensing unit), but the adjustment logic is preset. Through the fusion strategy after weight adjustment, the processing module 104 can more accurately update the yaw and translation information of the remote control module relative to the display terminal. The update process typically involves a state estimation loop, where yaw and translation information, as part of the displayed system state, are first predicted based on the motion model, and then corrected and updated using weighted fused sensor measurements. This update mechanism ensures that the yaw and translation information continuously reflects the true attitude and position of the remote control module, thus providing a stable and accurate estimate of the remote control module's attitude and position.

[0026] This application employs a preset weighted fusion strategy through processing module 104 to adjust the weights of the position information generated by the event camera and the attitude data sensed by the inertial sensing unit. This adjustment mechanism allows the display system to dynamically or statically optimize the data fusion process based on the real-time reliability or preset priority of different data. For example, when the infrared light spot sensed by the event camera is clear and has high confidence, the position information can be given a higher weight to fully utilize its high-precision positioning advantage; conversely, when the rapid movement of the remote control module makes the instantaneous attitude data of the inertial sensing unit more valuable, the weight of the attitude data can be appropriately increased to ensure timely response. This intelligent weight adjustment improves the accuracy and robustness of the yaw and translation information calculations of the remote control module 103 relative to the display terminal 101. Ultimately, this makes the determination of the intersection coordinates of the optical axis and the display plane of the display terminal 101 more accurate and stable, thereby achieving smoother, more precise, and more stable control of the cursor on the user interface of the display terminal 101, improving the user experience and system reliability.

[0027] In some embodiments of this application, the specific type of infrared light-emitting device selected in the infrared marking module and its layout on the display terminal have a significant impact on ensuring stable emission of infrared light spots, improving the accuracy of event camera sensing, and optimizing system integration. Improper selection or layout of the infrared light-emitting device may lead to unstable infrared signals, thereby affecting the positioning accuracy of the remote control module and the user experience.

[0028] Based on this, in one embodiment, the infrared light-emitting device is an infrared transistor; the infrared light-emitting device is disposed on the bezel of the display terminal.

[0029] Infrared transistors, typically referring to infrared light-emitting diodes (IR LEDs), are semiconductor devices that convert electrical energy into infrared light energy. They utilize the recombination effect of a PN junction; when forward biased, electrons and holes recombine at the PN junction, releasing energy and emitting infrared light in the form of photons. Infrared transistors can emit infrared light of specific wavelengths, aiding event cameras in efficient and accurate sensing, reducing ambient light interference, and thus improving the accuracy and confidence of infrared spot location information. The ease of driving and controlling infrared transistors also facilitates subsequent frequency or phase encoding, making it easier to identify different infrared light-emitting devices. Furthermore, placing infrared light-emitting devices on the bezel of the display terminal allows them to be integrated or fixed in the non-display area surrounding the screen. This arrangement offers several advantages. First, the bezel area usually has sufficient space to accommodate the infrared light-emitting devices without obstructing the display, maintaining the visual integrity of the display terminal. Second, placing the infrared light-emitting devices on the bezel ensures they are within the effective field of view of the event camera's optical axis, providing the event camera with clear, unobstructed infrared spot signals. Furthermore, the fixed positions on the frame help to form a stable geometric reference, enabling the processing module to more accurately calculate the yaw and translation information of the remote control module relative to the display terminal. By reasonably distributing at least two infrared emitting devices on the frame, a stable infrared marker array can be formed, providing a reliable positioning reference for the remote control module.

[0030] As an example, Figure 2 This is an exemplary schematic diagram of an infrared light-emitting device in an embodiment of this application, such as... Figure 2As shown, the inertial sensing unit can be denoted as IMU; the event camera can be denoted as EVS; and at least two infrared emitting devices, such as infrared LEDs, or simply infrared dots, are installed at fixed positions near the screen as a spatial reference. The two infrared dots have a known relative geometric relationship (e.g., baseline length, relative height, etc.). The infrared emitting devices can be integrated into all areas outside the screen and at the edges, for example, on both sides of the top bezel of the screen; separate "infrared strips" or "reference strips" placed above or below the screen; or on the set-top box or speaker strip, as long as their position relative to the screen is known.

[0031] This application, by specifically defining the infrared light-emitting device as an infrared transistor and placing it on the bezel of the display terminal, effectively solves the problems of infrared signal stability, sensing accuracy, and system integration. As a light-emitting element, the infrared transistor provides a stable, efficient, and easily controllable infrared light source. Its specific wavelength helps the event camera accurately capture infrared light spots in complex environments and generate high-quality position and confidence information. Simultaneously, placing the infrared transistor on the bezel of the display terminal not only fully utilizes the existing structure of the display terminal and avoids interference with the display area, but also ensures the visibility and geometric stability of the infrared light spots in the event camera's field of view. This provides a reliable positioning reference for the processing module, making the determination of the yaw and translation information of the remote control module based on position, confidence, and attitude data more accurate. This improves the calculation accuracy of the intersection coordinates of the optical axis and the display plane of the display terminal, ultimately achieving more stable and precise control of the cursor on the display terminal's user interface, thus enhancing the user experience.

[0032] In some embodiments of this application, the data transmission and power supply management of the components inside the remote control module are not clearly defined, which may lead to low data transmission efficiency or improper power consumption management, affecting the overall performance of the display system and user experience.

[0033] Based on this, in one embodiment, the remote control module 103 further includes a communication module and a control circuit; the control circuit is connected to the inertial sensing unit, the event camera and the communication module respectively; the communication module is used to transmit intersection coordinates or attitude data to the display terminal; the control circuit is used to provide power management and timing control for the inertial sensing unit, the event camera and the communication module.

[0034] The communication module is responsible for data exchange between the remote control module 103 and the display terminal 101. The communication module can be implemented using wireless communication technologies, such as Bluetooth, Wi-Fi, or radio frequency (RF). Bluetooth modules are typically suitable for low-power, short-range data transmission scenarios; Wi-Fi modules offer higher data transmission rates and wider coverage; while RF modules can be used for specific point-to-point communication needs. The choice of communication module is optimized based on the data transmission volume, transmission distance, and power consumption requirements between the remote control module and the display terminal. The main function of the communication module is to transmit key data generated or sensed internally by the remote control module—namely, the coordinates of the intersection of the optical axis of the event camera and the display plane of the display terminal, or the attitude data of the remote control module sensed by the inertial sensing unit—to the display terminal. The communication module receives data from the control circuit, converts it into a signal format suitable for wireless transmission, and transmits it through the antenna. The display terminal is equipped with a corresponding receiving module to receive and decode the data to achieve real-time, precise control of the cursor.

[0035] As the core management unit of the remote control module, the control circuit is responsible for coordinating the operation of various functional components within the module and providing necessary power management and timing control. The control circuit can be a microcontroller (MCU), integrating a processing module, memory, and various peripheral interfaces, capable of running preset firmware to manage the overall operation of the remote control module. The control circuit connects to the inertial sensing unit, event camera, and communication module via physical connections and logical interfaces. Physical connections are typically achieved through wires, flexible printed circuit boards (FPCs), or integrated packages to ensure the transmission of electrical signals. The control circuit is responsible for providing a stable power supply and precise timing for the inertial sensing unit, event camera, and communication module. In terms of power management, the control circuit can dynamically adjust the supply voltage and current according to the operating status of each component (e.g., sensor data acquisition, data transmission, etc.) to optimize power consumption and extend the battery life of the remote control module. In terms of timing control, the control circuit provides a precise clock signal through an internal clock or an external crystal oscillator, coordinating the data acquisition frequency of the inertial sensing unit, the data frame rate of the event camera, and the data transmission interval of the communication module to ensure data synchronization between components and real-time system response, thereby guaranteeing the accuracy and consistency of the remote control module's output data.

[0036] As an example, the display terminal 101 can be a television or a computer device; the communication module can be a wireless communication module that supports Bluetooth, 2.4GHz private radio frequency, Wi-Fi or infrared communication, and sends the pointing coordinates to the display terminal 101.

[0037] This application achieves effective management of the internal components of the remote control module and efficient data transmission with the display terminal by incorporating a communication module and control circuit within the remote control module. Specifically, the communication module reliably transmits the intersection coordinates or attitude data sensed or calculated by the remote control module to the display terminal, ensuring timely response to cursor control commands. Simultaneously, the control circuit provides unified power management and precise timing control for the inertial sensing unit, event camera, and communication module. This not only optimizes the power consumption of the remote control module and extends its lifespan but also ensures data synchronization and accuracy. This allows the processing module to determine yaw information, translation information, and intersection coordinates based on more stable and accurate data, ultimately improving the smoothness and accuracy of cursor control on the display terminal's user interface and avoiding cursor drift or delay issues caused by improper data transmission or internal management.

[0038] In some embodiments of this application, the event camera may be unable to consistently sense a sufficient number of infrared light-emitting devices due to factors such as obstruction, excessive distance, or ambient light, resulting in unstable or interrupted acquisition of vision-based position information, which in turn affects the accuracy and continuity of cursor control.

[0039] Based on this, in one embodiment, the processing module 104 is further configured to: when the number of infrared light-emitting devices sensed by the event camera is less than two, switch the control system to attitude control mode to control the cursor based on the attitude data sensed by the inertial sensing unit; when the number of infrared light-emitting devices sensed by the event camera is greater than or equal to two, control the display system to switch to a collaborative working mode of the event camera and the inertial sensing unit, so that the display system performs error correction based on the position information generated by the event camera and the attitude data sensed by the inertial sensing unit, updates the yaw information and translation information, and avoids sudden cursor changes.

[0040] When the processing module 104 detects that the number of infrared emitters sensed by the event camera is insufficient for reliable visual positioning (e.g., less than two), the display system automatically switches to attitude control mode. In this mode, the system no longer relies on the position information of the event camera, but controls the cursor entirely based on the attitude data of the remote control module sensed by the inertial sensing unit. The inertial sensing unit typically includes sensors such as accelerometers, gyroscopes, and magnetometers, which can provide real-time attitude information such as pitch, roll, and yaw of the remote control module. By mapping attitude changes to cursor movement on the user interface of the display terminal—for example, changes in the pitch angle of the remote control module control the vertical movement of the cursor, and changes in the yaw angle control the horizontal movement of the cursor—a backup, continuous cursor control method is provided when visual positioning fails.

[0041] When the processing module 104 detects that the event camera can sense two or more infrared emitting devices, the display system enters a cooperative working mode of the event camera and the inertial sensing unit. In this mode, the event camera provides high-precision position information of the infrared emitting devices, which can be used to accurately calculate the absolute position and attitude of the remote control module relative to the display terminal. Simultaneously, the inertial sensing unit provides high-frequency attitude data, but its position and attitude estimation may have accumulated errors (drift). The precise position information provided by the event camera can be used to correct the accumulated drift errors in the attitude data of the inertial sensing unit, thereby updating the yaw and translation information of the remote control module.

[0042] This application enables the display system to intelligently switch to attitude control mode when the event camera fails to detect a sufficient number of infrared emitters due to environmental factors. This ensures the continuity of cursor control and prevents the user from completely losing control of the cursor. Simultaneously, when the event camera provides sufficient visual information, the display system enters a collaborative working mode. It fuses the high-precision position information provided by the event camera with the high-frequency attitude data provided by the inertial sensing unit. Through an error correction mechanism, it effectively suppresses the drift error of the inertial sensing unit and compensates for any instantaneous data fluctuations that may occur in the event camera. This ensures that the yaw and translation information of the remote control module is continuously and accurately updated, improving the smoothness and stability of cursor control, preventing sudden cursor changes, and enhancing the fluency and accuracy of user operation.

[0043] In some embodiments of this application, when multiple infrared light-emitting devices are provided on the display terminal, if the infrared light spots emitted by the infrared light-emitting devices do not have clear distinguishing marks, the event camera may have difficulty accurately identifying the specific infrared light-emitting device corresponding to each light spot when it senses multiple light spots. This may lead to confusion between the position information and the actual device, thereby affecting the accuracy of the attitude data of the remote control module and potentially causing instability in cursor control.

[0044] Based on this, in one embodiment, the infrared light-emitting device flashes by frequency encoding or phase encoding; the event camera is also used to identify the infrared light-emitting device based on frequency encoding or phase encoding for identity recognition.

[0045] In this configuration, when infrared emitters flash using frequency encoding, each emitter can be configured to emit infrared light at a unique, preset flashing frequency. For example, this can be achieved by equipping each emitter with an independent oscillation circuit or by time-division multiplexing control from the main control unit, allowing them to periodically turn on and off at a specific frequency. When infrared emitters flash using phase encoding, all emitters can flash at the same reference frequency, but each emitter's flashing period has a preset phase difference relative to a common reference signal or between itself and other emitters. This can be achieved by precisely controlling the driving timing of each emitter, ensuring that it emits infrared light pulses at specific times. Based on this, after sensing the flashing infrared light spots, the event camera can capture the time event sequence of each spot. The event camera or its connected processing module can analyze the event sequence, for example, by extracting the flashing frequency of each light spot using spectral analysis methods such as Fourier transform, or by identifying its phase characteristics through time-series analysis. By comparing the detected frequency or phase with the pre-stored encoded information corresponding to each infrared emitter, the event camera can identify each infrared emitter, thereby achieving identification of different infrared emitters.

[0046] In this application, each infrared emitter has a unique identifier, enabling the event camera to accurately identify the specific infrared emitter corresponding to each infrared spot when simultaneously sensing multiple infrared light points. This solves the potential confusion problem between multiple infrared light points and ensures that the position information of each infrared emitter can be correctly associated with its actual identity. Therefore, the processing module significantly improves the accuracy and reliability of its calculations when determining the yaw and translation information of the remote control module relative to the display terminal based on accurate position information, confidence information, and attitude data. Ultimately, this allows for more precise determination of the intersection coordinates of the optical axis and the display plane of the display terminal, thereby ensuring the stability and accuracy of cursor control on the display terminal's user interface and greatly improving the smoothness and user experience.

[0047] In some embodiments of this application, the physical location and integration method of the processing module have a significant impact on the overall system performance, response speed, and manufacturing cost. If the processing module exists as a separate unit, or if its communication link with the remote control module or display terminal is long, it may introduce additional communication delays, increase system complexity, and potentially reduce the real-time performance of cursor control.

[0048] Based on this, in one embodiment, the processing module 104 is integrated into the remote control module or integrated into the motherboard of the display terminal.

[0049] The processing module 104 can be integrated into the remote control module. The processing components responsible for core tasks such as calculating yaw information, translation information, and intersection coordinates are directly embedded in the internal circuitry of the remote control module. For example, a high-performance microcontroller (MCU) or system-on-a-chip (SoC) can be integrated on the printed circuit board (PCB) of the remote control module. This MCU or SoC can directly receive data from the inertial sensing unit and the event camera and process it in real time. This integration method makes the remote control module a relatively independent intelligent unit capable of autonomously completing most data processing tasks. Alternatively, the processing module can be integrated onto the mainboard of the display terminal. The processing module serves as part of the core processing module of the display terminal, for example, utilizing the existing main processor of the display terminal or configuring a dedicated coprocessor for it. The remote control module transmits the position information and confidence information sensed by the event camera and the attitude data sensed by the inertial sensing unit to the display terminal via wireless communication (e.g., Bluetooth, Wi-Fi). After receiving the data, the processing module on the display terminal performs subsequent calculations and cursor control. This fully utilizes the typically more powerful processing capabilities and more stable power supply of the display terminal.

[0050] This application integrates the processing module into the remote control module or onto the motherboard of the display terminal, effectively solving the latency and complexity issues that may arise from the independent existence of the processing module or excessively long communication links. When the processing module is integrated into the remote control module, the data transmission path can be significantly shortened, communication latency reduced, thereby improving the real-time performance and response speed of cursor control, making user operation smoother. Simultaneously, the tight integration within the remote control module also helps simplify system wiring and reduce overall power consumption. When the processing module is integrated onto the motherboard of the display terminal, the more powerful processing capabilities and more stable power supply of the display terminal can be fully utilized, thereby simplifying the design of the remote control module, reducing its cost and power consumption, and potentially enabling more complex algorithm processing while ensuring computational accuracy. Both integration methods optimize the overall architecture of the display system, improve the integration, stability, and user experience of the display system, making cursor control more precise and efficient.

[0051] In some embodiments of this application, the coordinates of the intersection point may be obtained in the coordinate system of the display plane itself, which differs from the coordinate system used for cursor control inside the display system. This makes it impossible to directly use the cursor to accurately control the user interface of the display terminal, and further conversion and adaptation are required.

[0052] Based on this, in one embodiment, the intersection coordinates are coordinates in the coordinate system where the display plane is located; the processing module 104 is also used to convert the intersection coordinates into cursor coordinates of the display system 100, and transmit the cursor coordinates to the display terminal 101 to control the cursor of the user interface.

[0053] Specifically, after receiving the intersection coordinates in the display plane coordinate system, the processing module 104 needs to map them to the coordinate system used for cursor display within the display system. This includes operations such as translation, scaling, and possible rotation of the coordinate system. If the cursor coordinates in the display system are pixel coordinates, a proportional conversion needs to be performed based on the display terminal's resolution and the physical size of the display plane. This conversion process can be pre-stored in the mapping relationship or performed in real-time using a preset calculation formula. The processing module transmits the cursor coordinates to the display terminal to control the cursor in the user interface. The converted cursor coordinates are in a standardized format that the display terminal can directly understand and use. The processing module sends the standardized cursor coordinates to the display terminal, and after receiving the coordinates, the display terminal's operating system or display driver can precisely move the cursor on the user interface to the specified position.

[0054] As an example, the display system 100 can be a screen pointing system; the display terminal 101 can be, for example, a TV motherboard, a set-top box, a PC host, or a smart projector; the infrared marker module 102 can be an infrared LED, which can be referred to as a marker; it is installed in a fixed position near the screen and used as a spatial reference; the remote control module 103 can be a remote control, which can be used handheld; the processing module 104 can be an MCU, SoC, or FPGA; the inertial sensing unit can be a six-axis gyroscope, which can be referred to as an IMU; the event camera, which can be referred to as an EVS, has its optical axis facing the screen direction and is used to sense the infrared light spots emitted by the external marker.

[0055] Figure 3 This is a schematic diagram of the working process of the infrared transistor in the embodiments of this application; Figure 4 This is a schematic diagram illustrating the workflow of the event camera and inertial sensing unit in an embodiment of this application; as shown... Figure 3 and Figure 4 As shown, the screen pointing system includes at least two markers with fixed spatial positions to provide a spatial reference for the screen position; a remote controller with a built-in event camera and IMU, rigidly integrated. A processing module is used to obtain the pointing point in the screen coordinate system based on the marker image position and IMU attitude information; a communication module sends the pointing point to the display host; wherein, the EVS is responsible for observing the marker in the remote controller; the IMU provides the remote controller's attitude; the two are fused in the processing module to output the absolute coordinates on the screen plane; the process of EVS and IMU synchronization and fusion is as follows: EVS events are filtered by time window and clustered to obtain the marker image position. The IMU data is used for attitude calculation. The marker image position and attitude results are geometrically fused under a unified time basis to obtain the pointing ray and intersect it with the screen plane.

[0056] This application clarifies that the intersection coordinates are coordinates in the coordinate system of the display plane, and converts them into cursor coordinates that the display system can directly recognize and use, thus solving the incompatibility problem between different coordinate systems. This ensures the accuracy of the remote control module's pointing operation, enabling the cursor to be accurately positioned at the user's desired screen location, thereby improving the user interface's interactive experience and operational smoothness.

[0057] In some implementations, when the infrared marking module is equipped with multiple infrared emitters, effectively selecting the appropriate emitter for calculation to ensure accuracy and stability is a problem that needs to be addressed. Indiscriminately selecting any two emitters for calculation may introduce unnecessary errors and affect the precision of cursor control.

[0058] Based on this, in one embodiment, the number of infrared light-emitting devices is greater than two; the processing module 104 is further configured to: determine the priority of each infrared light-emitting device, determine two target infrared light-emitting devices according to the priority; and determine the yaw information and translation information of the remote control module relative to the display terminal based on the position information, confidence information and attitude data of the target infrared light-emitting devices.

[0059] Specifically, having more than two infrared emitters provides redundant infrared light sources, improving system robustness. Even if some infrared emitters are blocked or malfunction, the display system can still function normally. It also allows for the selection of the optimal infrared emitters for positioning calculations. The processing module 104 can evaluate the quality of each infrared emitter based on the signal strength, brightness, sharpness, or confidence information sensed by the event camera, thereby determining the priority of each emitter. Infrared spots with high signal strength, moderate brightness, sharp edges, and high confidence typically have higher priority. Furthermore, priority can also be determined based on the physical distribution of the infrared emitters on the display terminal. For example, those that provide a greater baseline distance can be prioritized to improve the accuracy of triangulation or positioning. After determining the priority of each infrared emitter, the processing module selects the two highest-priority infrared emitters from all sensed infrared emitters as target infrared emitters. Alternatively, while considering priority, spatial distribution factors can also be taken into account, selecting two high-priority infrared emitters that are spatially far apart to ensure the geometric accuracy of the positioning calculation. Once the two target infrared emitters are identified, the processing module 104 uses the position and confidence information of the two target infrared emitters, combined with the attitude data of the remote control module sensed by the inertial sensing unit, to accurately calculate the yaw and translation information of the remote control module relative to the display terminal. This process can employ triangulation, algorithms, or other pose estimation methods.

[0060] This application enables the processing module to intelligently determine the priority of each infrared emitter when the infrared marking module has more than two infrared emitters, and selects the two optimal target infrared emitters for subsequent yaw and translation information calculations. This avoids using infrared spot data that may contain noise or have low reliability, thereby improving the calculation accuracy and stability of yaw and translation information of the remote control module relative to the display terminal. By utilizing redundant infrared emitters and optimizing their selection, the display system can more robustly cope with environmental changes or performance degradation of some components, ensuring the smoothness and accuracy of cursor control and improving the user experience.

[0061] In some implementations, the user interface of the display terminal often has a complex layout and interaction logic, such as containing multiple functional areas, buttons, or menu items. In this case, simply mapping the intersection coordinates directly to the cursor position may result in inaccurate cursor positioning or failure to fully utilize the specific interactive features of the user interface, thereby affecting the convenience and accuracy of user operation.

[0062] Based on this, in one embodiment, the processing module 104 pre-stores a display mapping table, which includes the association between the display area of ​​the display terminal 101 and the cursor; the processing module is also used to: control the cursor to be displayed at the corresponding position on the display screen according to the intersection coordinates and the display mapping table.

[0063] Specifically, a display mapping table is a data structure that stores the association rules between the display area of ​​the display terminal and the corresponding position or behavior of the cursor on the display screen. This mapping table can be implemented in various forms; for example, it can be a lookup table containing the correspondence between specific physical coordinate ranges on the display terminal and the logical coordinates of the cursor on the user interface; or it can be a configuration file defining the mapping rules between different display areas (such as menu areas, content browsing areas, control button areas, etc.) and cursor behavior or position. Pre-storage means that the display mapping table is loaded into the processing module's memory during system startup or initialization so that it can be quickly looked up and used during cursor control.

[0064] The display mapping table contains the association between the display area of ​​the display terminal and the cursor. This can be a correspondence between the physical display area of ​​the display terminal (e.g., a range of pixel coordinates on the screen, a specific user interface element area) and the logical position or behavior of the cursor on the user interface. This association can be non-linear; for example, a small area in the upper left corner of the display terminal might be mapped to the cursor being on a specific menu item, while a large area in the center of the display terminal might correspond to the cursor moving freely within the content browsing area. Furthermore, this association can define the cursor sensitivity, acceleration curve, or cursor snapping effect in different display areas to optimize the user experience.

[0065] The processing module 104 is also used to control the cursor to be displayed at the corresponding position on the display screen based on the intersection coordinates and the display mapping table. Specifically, after receiving the intersection coordinates determined collaboratively by the event camera and the inertial sensing unit, the processing module does not directly use these coordinates as the final position of the cursor. Instead, it first uses a pre-stored display mapping table for conversion. The processing module queries the display mapping table, finds its corresponding position in the display area of ​​the display terminal based on the current intersection coordinates, and determines the final logical position of the cursor on the display screen according to the relationships defined in the mapping table. For example, if the intersection coordinates fall on a specific button area of ​​the display terminal, the display mapping table may directly position the cursor to the center of that button or trigger the button's selected state, thereby achieving more precise and intelligent cursor control.

[0066] This application utilizes a display system to convert the original intersection coordinates into a cursor position that better matches the user interface design of the display terminal. This allows cursor control to move beyond simple physical coordinate mapping and instead enable customized mapping based on the characteristics of the display area (such as menus, buttons, and content areas). For example, in specific interactive areas, the cursor can be precisely positioned on preset user interface elements, or different movement sensitivities can be applied to different areas, significantly improving the accuracy of cursor positioning and the smoothness and intuitiveness of user interaction. This avoids cursor positioning deviations or jumps on complex interfaces, greatly optimizing the user experience and enabling users to operate the display terminal more conveniently and accurately. This application also proposes a display method using the aforementioned display system. The method includes sensing an infrared light spot emitted by an infrared light-emitting device; generating position information and confidence information of the infrared light-emitting device based on the infrared light spot; sensing attitude data of a remote control module; determining yaw and translation information of the remote control module relative to the display terminal based on the position information, confidence information, and attitude data; determining the intersection coordinates of the optical axis of the event camera and the display plane of the display terminal based on the yaw information, translation information, and attitude data; and controlling the cursor of the user interface of the display terminal based on the intersection coordinates.

[0067] The details of the display method can be found in the previous description of the display system, and will not be repeated here.

[0068] The display system and display method provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the technical solutions and core ideas of this application. Those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. The above modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A display system, characterized by, include: Display terminal, infrared marking module, remote control module, and processing module; The infrared marking module includes at least two infrared emitting devices positioned based on the location of the display terminal; the remote control module includes an inertial sensing unit and an event camera, the optical axis of which faces the display terminal. The event camera is used to sense the infrared light spots emitted by the infrared light-emitting device, and to generate the position information and confidence information of the infrared light-emitting device based on the infrared light spots; The inertial sensing unit is used to sense the attitude data of the remote control module; The processing module is configured to determine the yaw and translation information of the remote control module relative to the display terminal based on the position information, the confidence information, and the attitude data; determine the intersection coordinates of the optical axis and the display plane of the display terminal based on the yaw information, the translation information, and the attitude data; and control the cursor of the user interface of the display terminal based on the intersection coordinates.

2. The display system according to claim 1, characterized in that, The processing module is further configured to: The weights of the position information and the attitude data are adjusted using a preset weight fusion strategy to update the yaw information and the translation information.

3. The display system according to claim 1, characterized in that, The infrared light-emitting device is an infrared transistor; the infrared light-emitting device is disposed on the bezel of the display terminal.

4. The display system according to claim 1, characterized in that, The remote control module further includes a communication module and a control circuit; the control circuit is connected to the inertial sensing unit, the event camera, and the communication module, respectively. The communication module is used to transmit the intersection coordinates or the attitude data to the display terminal; The control circuit is used to provide power management and timing control for the inertial sensing unit, the event camera, and the communication module.

5. The display system according to claim 1, characterized in that, The processing module is further configured to: When the number of infrared light-emitting devices sensed by the event camera is less than two, the system is controlled to switch to attitude control mode to control the cursor based on the attitude data sensed by the inertial sensing unit. When the number of infrared light-emitting devices sensed by the event camera is greater than or equal to two, the display system is controlled to switch to the collaborative working mode of the event camera and the inertial sensing unit, so that the display system performs error correction based on the position information generated by the event camera and the attitude data sensed by the inertial sensing unit, updates the yaw information and the translation information, and avoids the cursor abrupt change.

6. The display system according to claim 1, characterized in that, The infrared light-emitting device flashes using frequency encoding or phase encoding. The event camera is also used to identify the infrared light-emitting device based on the frequency encoding or the phase encoding, so as to perform identity recognition.

7. The display system according to claim 1, characterized in that, The processing module is integrated into the remote control module or onto the main board of the display terminal.

8. The display system according to claim 1, characterized in that, The coordinates of the intersection point are the coordinates in the coordinate system of the display plane. The processing module is further configured to convert the intersection coordinates into cursor coordinates of the display system, and transmit the cursor coordinates to the display terminal to control the cursor of the user interface.

9. The display system according to claim 1, characterized in that, The number of infrared light-emitting devices is greater than two; the processing module is also used for: The priority of each infrared emitting device is determined, and two target infrared emitting devices are identified based on the priority. The yaw and translation information of the remote control module relative to the display terminal are determined based on the position information, confidence information, and attitude data of the target infrared emitting devices.

10. The display system according to any one of claims 1-9, characterized in that, The processing module pre-stores a display mapping table, which includes the association between the display area of ​​the display terminal and the cursor; the processing module is further configured to: The cursor is controlled to be displayed at the corresponding position on the display screen based on the intersection coordinates and the display mapping table.

11. A display method, characterized in that, The method of using the display system according to any one of claims 1-10 includes: The infrared light spot emitted by the infrared light-emitting device is sensed, and the position information and confidence information of the infrared light-emitting device are generated based on the infrared light spot; Sensing attitude data from the remote control module; Based on the location information, the confidence information, and the attitude data, the yaw and translation information of the remote control module relative to the display terminal are determined. The intersection coordinates of the optical axis of the event camera and the display plane of the display terminal are determined according to the yaw information, the translation information, and the attitude data. The cursor of the user interface of the display terminal is controlled according to the intersection coordinates.

Images