Display devices and cursor display methods

Abstract

This application relates to the field of display technology, and provides a display device and a cursor display method to improve cursor display effect. In this application, the display device has a monitor configured to display a user interface; a controller maintains a cursor drawing channel independent of the system's native drawing process, and the controller is configured to: generate a software vertical synchronization signal synchronized with the monitor's screen refresh cycle; when the screen refresh cycle changes, determine the first moment of the next screen refresh based on the current screen refresh cycle; at a second moment before the first moment, in response to the software vertical synchronization signal, calculate the cursor prediction position at the first moment using a motion trajectory prediction algorithm based on interactive input data from the remote control; draw a target cursor image based on the cursor prediction position, and draw it to the monitor's frame buffer via the cursor drawing channel, so that the cursor is drawn on the user interface at the first moment and displayed at the cursor prediction position.

Description

Technical Field

[0001] This application relates to the field of display technology, and in particular to a display device, a cursor display method, a storage medium, and a computer program product. Background Technology

[0002] The display device can be controlled via a remote control. When using a remote control to operate the display device, a cursor for the remote control can be displayed on the user interface of the display device's screen to provide appropriate visual feedback to the user.

[0003] The remote control can be a pointing remote control. Pointing remote controls are interactive devices based on relative displacement detection. They use built-in sensors to sense their own movement in the air, which drives the corresponding cursor on the user interface to move accordingly, enabling airborne positioning and control of user interface elements. When a user points the pointing remote control at the monitor screen, a corresponding cursor (the pointing cursor) usually appears at that location on the user interface. This cursor moves with the pointing remote control; pointing remote controls provide users with a natural "what you see is what you point" interactive experience.

[0004] The remote control sometimes generates continuous, high-frequency displacement data, and the remote control refresh rate does not match the screen refresh rate. This can easily cause cursor jitter and unresponsiveness in the traditional native drawing process of the display device, resulting in high cursor display latency and difficulty in achieving a smooth "point-and-shoot" cursor effect, ultimately leading to poor cursor display quality. Summary of the Invention

[0005] Therefore, it is necessary to provide a display device and cursor display method, storage medium and computer program product to address the above-mentioned technical problems.

[0006] This application provides a display device, including:

[0007] The display is configured to: display a user interface; and independently display a cursor on the user interface, the position of which is determined by the position pointed to by the remote control in three-dimensional space;

[0008] A controller, coupled to the display and operating outside the display's operating system graphics synthesizer, maintains a cursor drawing channel independent of the system's native drawing process; the controller is configured to:

[0009] Receive interactive input data from the remote control;

[0010] Monitor the screen refresh cycle of the display and generate a software vertical synchronization signal that is synchronized with the screen refresh cycle;

[0011] If a change in the screen refresh cycle of the display is detected, the first moment of the next screen refresh of the display is determined based on the current screen refresh cycle of the display.

[0012] At a second moment before the first moment, in response to the software vertical synchronization signal, the cursor's predicted position at the first moment is calculated based on the interactive input data using a motion trajectory prediction algorithm.

[0013] The target cursor image of the remote control is drawn based on the cursor prediction position and drawn directly to the frame buffer of the display via the cursor drawing channel, so that the cursor is drawn on the user interface at the first moment and displayed at the cursor prediction position.

[0014] This application provides a cursor display method applied to a display device, the display device including a display and a controller; the display is configured to: display a user interface; and independently display a cursor on the user interface, the position of the cursor on the user interface being determined by the position pointed to by a remote control in three-dimensional space; the controller is coupled to the display and maintains a cursor drawing channel independent of the system's native drawing process, outside of the operating system's graphics synthesizer of the display;

[0015] The method includes:

[0016] Receive interactive input data from the remote control;

[0017] Monitor the screen refresh cycle of the display and generate a software vertical synchronization signal that is synchronized with the screen refresh cycle;

[0018] If a change in the screen refresh cycle of the display is detected, the first moment of the next screen refresh of the display is determined based on the current screen refresh cycle of the display.

[0019] At a second moment before the first moment, in response to the software vertical synchronization signal, the cursor's predicted position at the first moment is calculated based on the interactive input data using a motion trajectory prediction algorithm.

[0020] The target cursor image of the remote control is drawn based on the cursor prediction position and drawn directly to the frame buffer of the display via the cursor drawing channel, so that the cursor is drawn on the user interface at the first moment and displayed at the cursor prediction position.

[0021] This application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0022] This application provides a computer program product having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0023] The above technical solution has the following beneficial effects:

[0024] By maintaining a cursor drawing channel independent of the system's native drawing process, separate from the operating system's graphics compositor, and monitoring the display's screen refresh cycle to generate a software vertical synchronization signal synchronized with the refresh cycle, and then responding to this software vertical synchronization signal to perform cursor drawing and frame buffer writing at a second moment before the first moment, invalid over-rendering and screen tearing caused by data writing during frame buffer reading can be avoided. Furthermore, by responding to the software vertical synchronization signal, calculating the cursor's predicted position at the first moment based on interactive input data and using a motion trajectory prediction algorithm, and then drawing according to this predicted position, uneven and discrete remote control input events can be converted into a continuous and uniform display position sequence synchronized with the screen refresh cycle. This eliminates abrupt cursor step size changes and visual jitter caused by input interval fluctuations, avoids using outdated historical positions, and makes the cursor movement perceived by the user more closely resemble the actual control intent, improving the smoothness and responsiveness of pointing at the remote control. However, it is evident that by combining dynamic adaptation of the screen refresh cycle with cursor trajectory prediction, the problems of cursor lag and jitter can be alleviated, making the cursor display process smoother and more continuous, thereby improving the overall cursor display effect. By redetermining the first moment of the next screen refresh based on the current screen refresh cycle when a change in the screen refresh cycle is detected, and adjusting the cycle of the software vertical synchronization signal accordingly, the cursor drawing channel can automatically adapt to displays with different screen refresh rates (such as 50Hz, 60Hz, 120Hz, 240Hz, 288Hz, etc.) and dynamic switching scenarios without manual intervention or service restart, improving the compatibility and robustness of the solution under different hardware platforms and playback scenarios. By driving cursor drawing with the software vertical synchronization signal as the beat, rather than triggering drawing once for each remote control input event, one effective drawing can be performed within one screen refresh cycle, avoiding multiple invalid drawings caused by the remote control input frequency being higher than the screen refresh rate, and avoiding over-rendering. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application;

[0026] Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application;

[0027] Figure 3This is a schematic diagram of the software configuration of a display device provided in some embodiments of this application;

[0028] Figure 4 This is a schematic diagram of the system's native drawing process in one embodiment;

[0029] Figure 5 This is a flowchart illustrating a cursor drawing channel that is relatively independent of the screen refresh cycle in one embodiment.

[0030] Figure 6 This is a schematic diagram of the architecture of the input subsystem in one embodiment;

[0031] Figure 7 This is one of the flowcharts illustrating the steps performed by the controller in one embodiment;

[0032] Figure 8 This is one of the flowcharts illustrating cursor prediction in one embodiment;

[0033] Figure 9 This is a second schematic diagram of the cursor prediction process in one embodiment;

[0034] Figure 10 This is a flowchart illustrating the process of determining whether to draw a cursor image in one embodiment;

[0035] Figure 11 This is a flowchart illustrating the process of determining whether the screen refresh cycle has changed in one embodiment.

[0036] Figure 12 This is a schematic diagram of the structure of a display device in one embodiment;

[0037] Figure 13 This is a second schematic diagram of the steps performed by the controller in one embodiment;

[0038] Figure 14 This is a signaling interaction diagram of a cursor drawing method in one embodiment;

[0039] Figure 15 This is the third flowchart illustrating the steps performed by the controller in one embodiment;

[0040] Figure 16 A schematic diagram of a cursor drawing channel that incorporates a software vsync signal in one embodiment;

[0041] Figure 17 A schematic diagram of a cursor drawing channel in which a trajectory prediction strategy is introduced in one embodiment;

[0042] Figure 18 This is a flowchart illustrating cursor drawing in one embodiment. Detailed Implementation

[0043] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0044] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0045] In this embodiment, the display device 200 generally refers to a device with screen display and data processing capabilities. For example, the display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0046] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, users can operate the display device 200 via touch operation, mobile terminal 300, and control device 100. For example, control device 100 can be a remote control, stylus, gamepad, etc.

[0047] The mobile terminal 300 can function as a control device for human-computer interaction between the user and the display device 200. The mobile terminal 300 can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0048] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0049] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0050] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0051] In some embodiments, the detector 230 is used to collect signals from the external environment or to interact with the outside world. In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0052] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0053] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0054] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0055] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0056] In some embodiments, the audio output device 270 may be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. In some embodiments, the user input interface 280 may be used to receive instructions from user input.

[0057] The control device 100 is configured to control the display device 200 and to receive user input operation commands, converting them into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200. In some embodiments, the control device 100 can be a remote control. The remote control typically incorporates inertial measurement units such as gyroscopes, accelerometers, and magnetometers. Through these inertial measurement units or by employing infrared / optical pose positioning technology, the remote control can acquire its own attitude angles (e.g., yaw, pitch, roll angles) in three-dimensional space in real time. This data can be referred to as the remote control's positioning data. The remote control can transmit this positioning data to the display device via a wireless communication link (e.g., Bluetooth, infrared). The controller of the display device, using a coordinate mapping algorithm, can convert the positioning data sent by the remote control into a cursor position on the display's user interface, where the cursor can be displayed. Specifically, the remote control can be a pointing remote control.

[0058] In some embodiments, the display device 200 may run an operating system to enable user interaction. The operating system is a computer program used to manage and control the hardware and software resources of the display device 200. The operating system can provide a user interface, allowing users to interact with the display device 200 and supporting the execution of various applications. The operating system can be a native operating system based on a specific operating platform, a third-party operating system deeply customized for a specific operating platform, or a standalone operating system specifically developed for the display device.

[0059] An operating system can be divided into different modules or levels based on the functions it implements, for example... Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0060] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0061] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0062] like Figure 3 As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0063] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0064] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 3As shown, hardware drivers can be configured in the kernel layer. The kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, Wi-Fi driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver. The above example is merely a simple division of operating system functions and does not limit the specific operating system form of the display device 200 in this embodiment. Depending on the functions of the display device, the type of operating system, and other factors, the number and specific types of layers in the operating system can take other forms.

[0065] The following is a description of the technical terms used in this application:

[0066] The system native rendering process is the rendering process provided by the operating system of the display device; most application elements are usually drawn through the system native rendering process; the system native rendering process usually adopts the collaborative working mechanism of the operating system graphics compositor and controller (also known as hardware display controller, display controller).

[0067] An operating system graphics compositor is a system-level service or module running in the operating system kernel space or user space. It receives graphics layers (typically one graphics layer corresponds to one graphics buffer) submitted by multiple applications or system services, and blends these layers according to their display order, transparency, scaling, cropping, rotation, and other compositing parameters to generate a complete display image. This image is then written to a framebuffer for the controller to read and output to the screen. The operating system graphics compositor can also be called SurfaceFlinger, abbreviated as SF.

[0068] The following describes the specific processes involved in the system's native rendering workflow:

[0069] The first stage involves the generation of hardware vertical synchronization signals:

[0070] The controller scans the framebuffer (FB) at a certain refresh rate (e.g., 60Hz, 120Hz), reading pixel data from the framebuffer and outputting it to the display screen to show the image on the user interface. This refresh rate is also called the screen refresh rate. When the controller completes one frame scan (i.e., one complete full-screen scan), it generates a hardware vertical synchronization signal (vsync signal) via a hardware pin. The generation frequency of the hardware vsync signal is usually consistent with the screen refresh rate. vsync stands for Vertical Synchronization. The reciprocal of the screen refresh rate is the screen refresh cycle, which is the time interval required to read pixel data from the framebuffer for one frame and output it to the screen. For example, when the screen refresh rate is 60Hz, the screen refresh cycle is approximately 16.67ms; when the screen refresh rate is 120Hz, the screen refresh cycle is approximately 8.33ms. The screen refresh cycle is consistent with the period of the hardware vertical synchronization signal.

[0071] Display devices with different hardware platforms have different screen refresh rates. Some have a maximum refresh rate of 60Hz, while others have 120Hz, or even 240Hz or 288Hz. Furthermore, the screen refresh rate of a display device with a particular hardware platform can dynamically change in different scenarios. For example, in video playback, when playing PAL format (25fps / 50fps) video content, the screen refresh rate is switched to 50Hz to perfectly match the frame rate. In this case, the generation period of the hardware vsync signal (denoted as vsync_period) is approximately 20ms. When playing NTSC format (30fps / 60fps) video content, the screen refresh rate is switched to 60Hz, and the generation period of the hardware vsync signal (vsync_period) is approximately 16.67ms. It can be understood that the generation period of the hardware vsync signal (vsync_period) = 1 / the generation frequency of the hardware vsync signal. Among them, Hz stands for Hertz, ms stands for millisecond; fps stands for frames per second; PAL stands for Phase Alternating Line; NTSC stands for National Television System Committee.

[0072] The second stage involves the synthesis and synchronization of the operating system's graphics synthesizer.

[0073] After receiving the hardware vsync signal, the operating system's graphics compositor converts it into two software vsync signals (also known as software vertical synchronization signals): the APP vsync signal and the SF vsync signal. The APP vsync signal is primarily used to drive the application to start frame drawing according to the screen refresh rate, thereby drawing View objects (such as buttons, text, images, etc.) onto the graphics buffer corresponding to the layer. Here, APP stands for Application.

[0074] The operating system's graphics compositor sends the APP vsync signal across processes via Socket to the choreographer in the application process. Upon receiving the APP vsync signal, the choreographer triggers the application to draw the next frame's content to its layer's corresponding graphics buffer using a Handler mechanism. Specifically, for example... Figure 4 The APP vsync signal 1 triggers the application to draw image 0 to the graphics buffer corresponding to the layer; the APP vsync signal 2 triggers the application to draw image 1 to the graphics buffer corresponding to the layer. After drawing the images to the graphics buffers corresponding to the layers, the application submits the graphics buffers corresponding to the layers to the operating system's graphics compositor. The SF vsync signal can directly trigger the GPU or hardware compositor (HWComposer) to composite the data in the graphics buffers of all layers (including the application interface, status bar, cursor layer, etc.), and then write the composited data to the frame buffer.

[0075] Under normal circumstances, the application completes drawing and submits the graphics buffer within one vsync cycle; the GPU or hardware compositor completes compositing within the same vsync cycle. The display controller reads the frame buffer and outputs it to the screen at the beginning of the next vsync cycle. Therefore, from the time the application receives the APP vsync signal and begins drawing until the frame is displayed on the screen, two complete vsync cycles (approximately 33ms at 60Hz) are required. Figure 4 In the text, the time offset between the "Draw (APP)" row and the "Display (Screen)" row reflects this delay. For example, image 1 is drawn in the first vsync cycle and displayed in the third vsync cycle.

[0076] As mentioned above, the system's native drawing process has an inherent delay of at least two vsync cycles. If the cursor is drawn using the system's native drawing process, the cursor movement will be less responsive, making it difficult to meet the real-time requirements of pointing remote control interaction and achieve a smooth "point-and-shoot" cursor effect. This results in low processing efficiency and high latency.

[0077] To reduce cursor rendering latency, improve responsiveness, and achieve smooth, point-and-click cursor effects, related technologies utilize a cursor drawing channel independent of the system's native drawing process. This channel bypasses the drawing and compositing operations of the system's native drawing process, directly writing the cursor image's pixel data to the framebuffer via the framebuffer device interface. This bypasses the latency of at least two vsync cycles in the system's native drawing process, reducing the time from cursor input to display to less than one screen refresh cycle, thus improving cursor movement responsiveness and real-time reaction capabilities. The full name of the framebuffer device is Framebuffer Device, abbreviated as FbDev.

[0078] The cursor drawing channel is a logical concept, referring to the data processing path from data reception, processing, and prediction to writing the cursor image into the frame buffer. The cursor drawing channel is independent of the system's native drawing process and does not go through the operating system's graphics compositor.

[0079] The software component that implements the cursor drawing channel can be called the cursor drawing module; the cursor drawing module refers to the collection of code units that implement the cursor drawing channel, which may include the cursor management unit (CursorManager), the signal simulation unit (VsyncEmulator), the trajectory prediction unit (MotionPredictor), and the cursor drawing unit, etc.

[0080] The cursor management unit is the control component in the cursor drawing module. It is responsible for receiving external events (such as screen refresh cycle change notifications and software vertical synchronization signals), managing cursor states (such as show / hide and style switching), and coordinating the work between various units in the cursor drawing module. Specifically, the cursor management unit can send interactive input data from the remote control to the storage unit for storage; moreover, the cursor management unit can set up cursor event listeners and inject the received cursor events into the trajectory prediction unit; simultaneously, the cursor management unit can implement client logic for listening to vertical synchronization signal cycle changes (vsyncperiodChanged) to receive update notifications of the screen refresh cycle; in addition, the cursor management unit can also set up event listeners for simulated vertical synchronization signals. When a simulated vertical synchronization signal is received, it reads the cursor prediction position after trajectory prediction by the trajectory prediction unit, thereby enabling the cursor drawing unit to perform cursor drawing operations based on the cursor prediction position.

[0081] The signal simulation unit is primarily used to generate software vertical synchronization signals based on the screen refresh cycle. Internally, it can contain a high-precision timer capable of generating periodic software vertical synchronization signals according to a set period. The trajectory prediction unit is responsible for running motion trajectory prediction algorithms (such as linear extrapolation, Kalman filtering, etc.) based on the cursor's historical position, outputting the predicted cursor position at a future time. The trajectory prediction unit can receive instructions from the cursor management unit and return the predicted cursor position. The cursor drawing unit, which can be called by the cursor management unit, is responsible for performing cursor image drawing operations. This drawing operation may include generating a target cursor image and writing the pixel data of the target cursor image into the frame buffer corresponding to the predicted cursor position through the frame buffer device interface.

[0082] The remote control reports cursor coordinate data via HID (Human Interface Device) or the input subsystem. Each report of cursor coordinate data by the remote control is considered a remote control input event. In the cursor drawing channel, the cursor management unit immediately triggers cursor drawing whenever a remote control input event occurs. This "render on input" mode causes the cursor rendering frequency to follow the remote control's input frequency, but is relatively independent of the screen refresh rate (e.g., 16.67ms cycle for 60Hz). Furthermore, since the screen refresh rate dynamically changes with the scene, it cannot adaptively match the screen refresh rate, easily leading to uneven rendering, over-rendering, cursor trajectory jitter, or screen tearing, severely impacting the user experience.

[0083] Furthermore, the cursor drawing channel employs a dual-framebuffer mechanism, with one framebuffer serving as the current display framebuffer (Front Framebuffer) and the other as the current drawing framebuffer (Back Framebuffer). The current display framebuffer is the framebuffer being read, and its contents are currently displayed on the screen. The current drawing framebuffer is the framebuffer used to write new cursor data. After each cursor drawing operation, the cursor management unit immediately performs a pointer swap, making the recently written Back Framebuffer the new Front Framebuffer, while the original Front Framebuffer becomes the Back Framebuffer. The pointer swap operation can occur at any time during the framebuffer reading process. Because pointer swapping is triggered by random remote control input events and is independent of the hardware vsync signal, the following problems can easily occur under the double buffering mechanism: First, if a line of the original Front Framebuffer is being read when a swap occurs, the upper half of the screen will retain the old cursor content, while the lower half will switch to the new cursor content, causing screen tearing. Second, since each remote control input event triggers drawing and swapping, multiple swaps may occur within a single screen refresh cycle (e.g., 16.67ms), causing the content of the Front Framebuffer to be frequently overwritten. Ultimately, only the result of the last swap may be fully displayed, with multiple intermediate drawings being invalid over-rendering. Moreover, the direct discarding of intermediate frame data can easily cause the displayed cursor position to jump around, resulting in an uneven trajectory. In addition, remote control input events have inherent frequency fluctuations and transmission delays. When drawing the cursor, historical positions are usually used, which leads to cursor position lag and the responsiveness failing to meet user expectations.

[0084] Assume the remote control input frequency is 100fps with uneven intervals (minimum interval 7ms, maximum interval 13ms), and the screen refresh rate is 60Hz (screen refresh cycle approximately 16.67ms). Figure 5 As shown, the timing sequence is as follows:

[0085] Initially (before t1): The Front Framebuffer contains image 0, and the Back Framebuffer is empty.

[0086] At time t1: Input data p1 is received and written to the Back Framebuffer, then pointer swapping is performed. At this time, the Front Framebuffer corresponds to p1, and the Back Framebuffer corresponds to image 0. p1 is displayed on the screen at time t2.

[0087] During the period from t2 to t3, input data p2 is generated. p2 is written to the Back Framebuffer and a pointer swap is performed again. At this time, the Front Framebuffer corresponds to p2, and the Back Framebuffer corresponds to p1. p2 is displayed on the screen at time t3.

[0088] During the period from t3 to t4: Input data p3 and p4 are generated. The Front Framebuffer corresponds to p2, and the Back Framebuffer corresponds to p1. When p3 arrives, p3 is first written to the Back Framebuffer, and a pointer swap is performed. At this time, the Front Framebuffer corresponds to p3, and the Back Framebuffer corresponds to p2. Immediately afterwards, p4 arrives, p4 is written to the Back Framebuffer, and a pointer swap is performed. At this time, the Front Framebuffer corresponds to p4, and the Back Framebuffer corresponds to p3. At time t4, p4 is displayed on the screen, and p3 is not displayed.

[0089] During the period from t4 to t5: Input data p5 and p6 are generated. The Front Framebuffer corresponds to p4, and the Back Framebuffer corresponds to p3. When p5 arrives, it is written to the Back Framebuffer, and a pointer swap occurs. At this point, the Front Framebuffer corresponds to p5, and the Back Framebuffer corresponds to p4. Immediately afterwards, p6 arrives, is written to the Back Framebuffer, and another pointer swap occurs. At this point, the Front Framebuffer corresponds to p6, and the Back Framebuffer corresponds to p5. At time t5, p6 is displayed on the screen, and p5 is not displayed.

[0090] In the above process, because each remote control input event triggers drawing and pointer swapping, and the swapping timing is not synchronized with the screen refresh cycle, multiple remote control input events continuously overwrite the BackFramebuffer within a single screen refresh cycle. Ultimately, only the last written data is displayed (e.g., p4 overwrites p3, p6 overwrites p5). The actual display sequence is p1, p2, p4, p6, skipping intermediate frames such as p3 and p5. This abrupt change in step size causes cursor movement to be unresponsive and jittery, and the multiple redundant drawing and swapping within the same screen refresh cycle results in over-rendering. In some cases, it is desirable to achieve uniform display of input data (i.e., displaying p1 at time t2, p3 at time t3, and p5 at time t4).

[0091] Based on this, this application provides a remote control cursor adaptive rendering scheme built around "dynamic screen refresh rate synchronization + trajectory prediction compensation". This scheme, based on a cursor drawing channel independent of the system's native drawing process, obtains a software vertical synchronization signal synchronized with the screen refresh cycle. In the second moment before the first moment, it responds to this software vertical synchronization signal to perform cursor drawing and frame buffer writing, avoiding invalid over-rendering and screen tearing caused by data writing during frame buffer reading. Simultaneously, based on a motion trajectory prediction algorithm, this scheme calculates the cursor's predicted position at the first moment in response to the software vertical synchronization signal and draws according to this predicted position, transforming uneven and discrete remote control input events into a continuous and uniform display position sequence synchronized with the screen refresh cycle. This solution eliminates abrupt cursor step size changes and visual jitter caused by fluctuations in input intervals, avoids using outdated historical positions, thereby alleviating cursor position lag issues and improving responsiveness and interaction smoothness. Furthermore, the solution senses changes in screen refresh rate in real time. When a change in the screen refresh cycle is detected, it determines the first moment of the next screen refresh based on the current screen refresh cycle and adjusts the period of the software vertical synchronization signal accordingly. This allows the cursor drawing channel to automatically adapt to different screen refresh rates and dynamically switching scenarios without manual intervention or service restarts. It avoids problems such as invalid over-rendering due to refresh rate mismatch, improving the compatibility and robustness of the solution across different hardware platforms and playback scenarios.

[0092] The solution provided in this application can be applied to display devices, and mainly involves the following aspects:

[0093] I. Hardware Basics:

[0094] Display devices can include smart TVs with an operating system, which may or may not be limited to Android. Display devices include monitors that support multiple refresh rates or dynamic refresh rate switching, covering specifications such as 60Hz and 120Hz, compatible with scenario-based switching between 50Hz (PAL) and 60Hz (NTSC), and support hardware upgrades to 240Hz high refresh rates. Display devices can also have independent frame buffer hardware overlays for cursor drawing, dual frame buffer architectures, and FbDev hardware direct drawing capabilities. The display device's chip can support kernel-level input event processing, multi-process concurrency, and hardware rendering acceleration, meeting the real-time calculation requirements for vertical synchronization signal simulation and trajectory prediction.

[0095] The remote control for the display device can use infrared, Bluetooth, UWB (Ultra-Wideband), or other communication methods. The remote control supports spatial pointing coordinate acquisition and can output interactive actions such as clicking, swiping, and hovering. When the user performs different interactive actions such as clicking, swiping up, down, left, right, and hovering, the display can show different styles in real time, and provides cursor display, hiding, and changing interfaces. The remote control can cover multiple input frequency specifications such as 100fps, 120fps, and 240fps.

[0096] II. Software Basics:

[0097] The underlying system foundation may include an input subsystem, such as Figure 6 As shown, the input subsystem supports hardware devices including infrared remote controls, Bluetooth remote controls, pointing remote controls, keyboards, mice, and touchscreens. The kernel space of the input subsystem can shield the differences between various hardware devices, obtain data from them, and convert the data into standard input events. Each hardware device typically corresponds to a node under the ` / dev / input` directory. This node serves as the interface for user space to access the kernel input subsystem. User space can perform data reading, writing, and device control operations on this node using system calls such as `read`, `write`, and `ioctl` (Input / Output Control). ` / dev` is a virtual file system directory used to store device nodes corresponding to various hardware devices in the system. The user space of the input subsystem can access these nodes to process input events. In some types of operating systems (such as Android), input event processing is handled in the system service (system_server) process, which is mainly implemented by the input manager service (InputManagerService, IMS). The input manager service can include the input management submodule (InputManager), the input event collection submodule (InputReader), the input event dispatch submodule (InputDispatcher), and the direct rendering submodule (DirectRender).

[0098] The input management submodule is responsible for coordinating the work between various submodules within the input manager service. The input event collection submodule can include an event hub and a cursor input mapper. The event hub listens to each input device node under the ` / dev / input` directory, waiting for and reading raw input events through the epoll mechanism. The event hub shields the differences in underlying hardware devices, providing a unified event source to the input event collection submodule. The cursor input mapper is a mapper module within the input event collection submodule. It handles cursor-type devices such as mice, trackballs, and pointing remotes. The cursor input mapper converts raw cursor movement data (relative displacement or absolute position) into standard motion events (MotionEvents) and calculates the cursor's coordinates on the screen, button click states, and other information, which are then distributed to the application by the input event distribution submodule. The application process is responsible for processing the events distributed by the input event distribution submodule. These events include, but are not limited to, key events (KeyEvents) and the aforementioned motion events. Key events typically correspond to actions such as pressing / releasing buttons on remotes and keyboards, while motion events typically correspond to actions such as moving and swiping on touchscreens, mice, and pointing remotes. The application process can also set cursor styles, monitor cursor display status, and control whether the cursor is shown / hidden. The epoll mechanism provides an extensible I / O event notification mechanism for the kernel space.

[0099] To accelerate cursor rendering and reduce the load on system service processes, the cursor display function is separated into a dedicated cursor drawing process, primarily implemented by the aforementioned cursor drawing module. In addition to the aforementioned cursor management unit, signal simulation unit, trajectory prediction unit, and cursor drawing unit, the cursor drawing module may also include a native input reader and a cursor resource management unit. The native input reader collects cursor events from the remote control; the cursor resource management unit parses and decodes cursor images, which are cursor resources.

[0100] To address the issue of remote control swipe data not being transmitted via the traditional / dev / input node, the native input reader can use the / dev / hidrawX node to transmit raw HID data. The cursor management unit parses the swipe data from this node and interacts with the direct rendering submodule in the system service process via Binder IPC (Binder Inter-Process Communication) to simulate the parsed swipe events as standard motion events and inject them into the input manager service, thus completing event distribution. Furthermore, the system service process can provide interfaces such as init, onInjectHidMotionEvent, setAnimationDefaultApp, onCursorRendererChanged, setAnimationIconType, and getAnimationIconType for interaction with the cursor drawing module, including initialization and cursor animation style switching. The application process can also set the cursor style, control cursor display / hiding, and monitor cursor display status through corresponding interfaces. Specifically, the cursor drawing module calls the onInjectHidMotionEvent interface (HID motion event injection callback interface) to inject the remote control interaction input data parsed from the hidraw node into the system service process, converting it into standard motion events and distributing them to the application layer. When cursor rendering parameters in the cursor drawing module change (such as cursor position, style, or visibility state), the system service process can be notified through the callback of the onCursorRenderChanged interface (cursor rendering change callback interface) so that the system can synchronously update the cursor display state or perform other processing. The / dev / hidrawX node is the raw access node for HID devices; hidraw stands for Human Interface Device Raw, which translates to HID raw device.

[0101] The `init` function is the initialization interface, used to perform the initial configuration work of the cursor drawing module. `onInjectHidMotionEvent` is the HID motion event injection callback interface, triggered when the cursor drawing module injects HID sliding data via Binder. `setAnimationDefaultApp` is the interface for setting the default application of the animation, used to specify the default application for the cursor animation effect. `onCursorRendererChanged` is the cursor renderer change callback interface, called when the cursor rendering method or state changes. `setAnimationIconType` is the interface for setting the animation icon type, used to configure the icon style of the cursor animation (such as arrow, hand, loading, etc.). `getAnimationIconType` is the interface for getting the animation icon type, used to query the icon type of the current cursor animation. These interfaces can be located in the direct rendering submodule for interaction and configuration with the cursor drawing module.

[0102] III. Communication Basics:

[0103] End-side wireless communication includes infrared, Bluetooth, UWB, or other short-range wireless communication links between the remote control and the display device. This link can stably transmit interactive data such as pointing coordinates, button presses, and swipes, providing a data source for cursor position updates.

[0104] Intra-system cross-layer communication includes a device node communication mechanism between kernel space and user space. This mechanism uses the / dev / input node and the hidraw node to transfer input data from kernel space to user space.

[0105] Inter-process communication includes the Binder IPC mechanism and the AIDL (Android Interface Definition Language) interface specification. This mechanism enables cross-process data transmission and event callbacks between the cursor drawing service and the layer compositing service, as well as system service processes (such as IMS in this service), providing communication support for refresh rate monitoring and event injection.

[0106] IV. Basic Display Logic:

[0107] In the system's native rendering process, the operating system's graphics compositor generates a software vertical synchronization signal based on the hardware vertical synchronization signal, and uniformly schedules application rendering and layer compositing, thereby natively achieving refresh rate matching between the producer (application) and the consumer (operating system graphics compositor), avoiding rendering abnormalities caused by frame rate mismatch.

[0108] The cursor drawing channel adopts an independent frame buffer overlay drawing scheme, which has the basic capability of low-latency rendering, and supports real-time switching of cursor style with interactive actions and adaptive adaptation of coordinate resolution.

[0109] Dynamic screen refresh rate switching logic refers to the ability of display devices to dynamically switch screen refresh rates based on video format or playback scenario. This allows for automatic switching between different screen refresh rates, such as 50Hz and 60Hz, thus providing a basis for scene adaptation for adaptive rendering.

[0110] The cursor rendering channel features cursor resource parsing, style switching, display and hiding control, resolution adaptive matrix adaptation (e.g., 2K resolution to 4K resolution), and the ability to directly render by calling the frame buffer device interface. It can independently complete cursor rendering without relying on the operating system's graphics compositor. The specific process includes:

[0111] (1) Input event acquisition and transmission: The kernel layer receives input data from the remote control through the input subsystem, converts it into standard input events, and exposes it to the user space through the / dev / input node. In response to the problem that the remote control sliding data cannot be accessed through the input subsystem, it can be transmitted through the hidraw node. After being parsed by the cursor management unit, the event is injected into the input manager service through Binder IPC communication to complete the event distribution.

[0112] (2) Cursor rendering architecture: In order to improve rendering speed and responsiveness, the cursor drawing module is made into an independent dedicated service, which is separated from the system's native drawing process and the drawing and compositing process. It directly calls the frame buffer device interface based on the cursor position to complete the drawing. An independent Framebuffer overlay layer and dual buffer cache architecture are adopted to reduce the rendering link latency.

[0113] (3) Auxiliary adaptation capability: Supports resolution adaptation of cursor coordinates, and completes the adaptation of 1920×1080 basic coordinates to different screen resolutions such as 4K through transformation matrix; supports basic functions such as cursor style switching, display / hide, and resource parsing.

[0114] In one embodiment, this application provides a display device. The display device includes a display and a controller; wherein the display device is controlled by a remote controller.

[0115] The display is configured to show a user interface; and to display a cursor independently on the user interface, the position of which is determined by the position pointed to by the remote control in three-dimensional space.

[0116] The position that the remote control points to in three-dimensional space usually refers to the point where the three-dimensional direction line pointed to by the front end of the remote control (or the preset pointing axis) intersects the screen plane in a spatial coordinate system with the monitor screen as the reference; the two-dimensional coordinates of this point are used to control the position of the cursor on the screen.

[0117] The controller is coupled to the display; the controller maintains a cursor drawing channel independent of the system's native drawing process, outside of the display's operating system graphics synthesizer.

[0118] The operating system's graphics compositor and controller work together through a vertical synchronization signal to complete the system's native rendering process. Specifically, the controller scans the pixel data of the image from the frame buffer at a certain screen refresh rate and outputs it to the screen. After completing one frame scan, the controller generates a hardware vertical synchronization signal. The operating system's graphics compositor follows the vertical synchronization signal to complete the compositing of multiple graphics layers and buffer switching, and writes the pixel data of the composited image into the frame buffer. The pixel data in the frame buffer is used by the controller for the next scan.

[0119] In this case, the frame buffer written by the operating system graphics compositor and the frame buffer written by the cursor drawing channel are the same physical frame buffer; or, when the operating system of the display device has an independent hardware cursor layer, the cursor drawing channel can also write to a dedicated frame buffer, which is then overlaid and displayed by the controller.

[0120] The aforementioned controller is configured to execute Figure 7 The steps shown are as follows:

[0121] Step S701: Receive interactive input data from the remote control.

[0122] In some embodiments, interactive input data is a set of data generated and reported by the remote control in response to user operation, including at least one or more of the following: (1) cursor coordinate movement data: the displacement or absolute coordinates of the cursor on the screen calculated based on the spatial posture change of the remote control, used to control the cursor to follow the direction of the remote control; (2) button event data: including the key values ​​corresponding to the pressing, releasing or long-pressing actions of physical buttons (such as confirmation button, return button, menu button) on the remote control; (3) sliding / touch data: the sliding direction, distance, speed and other data generated by the remote control touchpad or sliding area, which can be mapped to scrolling or dragging operations on the screen; (4) hovering data: the duration and area of ​​the remote control briefly hovering at a certain pointing position, used to trigger hovering feedback or menu display.

[0123] Step S702: Monitor the screen refresh cycle of the display and generate a software vertical synchronization signal that is synchronized with the screen refresh cycle.

[0124] The cursor drawing module running on the controller can interface with the operating system's graphics compositor (such as SurfaceFlinger). The operating system's graphics compositor can obtain the screen refresh rate. By calling back to the aforementioned interface, the cursor drawing module can obtain the display's screen refresh rate in real time. Then, based on this screen refresh rate, the cursor drawing module can use a timer to generate a vertical synchronization signal that matches the screen refresh rate. This vertical synchronization signal is not generated by the controller based on hardware pins, and therefore can be called a software vertical synchronization signal. This software vertical synchronization signal is independently controlled by the cursor drawing module, but its period is consistent with the period of the hardware vsync signal. Through this software vertical synchronization signal, the cursor drawing module can transform the random rendering originally driven by remote control input events into uniform rendering driven by a fixed beat, thereby avoiding problems such as uneven rendering, over-rendering, jitter, and tearing caused by the mismatch between the input frequency and the screen refresh rate.

[0125] Furthermore, when the screen refresh rate changes dynamically due to system switching of display format or user settings (e.g., switching from 60Hz to 50Hz), the cursor drawing module can sense the change through a continuous monitoring mechanism and adjust the generation cycle of the software vertical synchronization signal accordingly, thereby achieving adaptive synchronization.

[0126] The above steps can obtain a rendering triggering cycle that is consistent with the generation cycle of the hardware vsync signal without modifying the operating system's graphics compositor itself, providing a time reference for subsequent trajectory prediction and periodic cursor drawing.

[0127] Step S703: If a change in the screen refresh cycle of the display is detected, the first moment of the next screen refresh of the display is determined based on the current screen refresh cycle of the display.

[0128] The monitor's screen refresh rate may change from one value to another due to system scene switching (such as switching video standards from PAL to NTSC), user manual settings, or dynamic hardware refresh rate adjustments. The cursor drawing module can monitor the interface established with the operating system's graphics compositor to obtain the current screen refresh rate in response to changes in the refresh rate. Based on the current screen refresh rate, the cursor drawing module calculates the time of the monitor's next screen refresh, which can be referred to as the first moment.

[0129] In step S704, at the second moment before the first moment, in response to the software vertical synchronization signal, the cursor prediction position at the first moment is calculated based on the interactive input data using a motion trajectory prediction algorithm.

[0130] The first moment is the time when the display will refresh next, calculated based on the current screen refresh cycle. The first moment usually corresponds to the rising edge of the next hardware vsync signal pulse. At the first moment, the controller will start reading the new frame image in the frame buffer and output it to the screen.

[0131] The cursor drawing module takes a certain moment between the current moment and the first moment as the second moment. There is a certain time offset between the second moment and the first moment (usually a few milliseconds). The second moment is mainly reserved for cursor drawing, that is, the second moment before the first moment. In response to the software vertical synchronization signal, it can reserve a certain amount of time for cursor drawing, so that the cursor can be successfully drawn on the user interface at the first moment and displayed at the cursor prediction position.

[0132] In response to the software vertical synchronization signal, when the second moment arrives, the cursor drawing module calculates the cursor's predicted position at the first moment based on the interactive input data received from the remote control using a motion trajectory prediction algorithm. This motion trajectory prediction algorithm can be an algorithm that estimates the cursor's future position based on historical interactive input data (including past cursor positions, timestamps, velocities, accelerations, etc.). The motion trajectory prediction algorithm can be, for example, but not limited to, a linear prediction algorithm (based on velocity), a second-order prediction algorithm (based on acceleration), or a Kalman filter algorithm. By invoking the motion trajectory prediction algorithm, the most likely position of the cursor at the upcoming first moment can be predicted in advance based on the received historical interactive input data, thus compensating for lag caused by remote control input cycle fluctuations, transmission delays, and other factors.

[0133] Step S705: Draw the target cursor image of the remote control based on the cursor prediction position, and draw it directly to the frame buffer of the display through the cursor drawing channel so that the cursor is drawn on the user interface at the first moment and displayed at the cursor prediction position.

[0134] The target cursor image of the remote control is the bitmap data corresponding to the cursor style to be displayed. After preprocessing such as decoding, scaling, and color conversion, the target cursor image is stored as a pixel array with a transparency channel.

[0135] The cursor drawing module can directly write the pixel data of the target cursor image into the address area corresponding to the predicted cursor position in the BackFramebuffer via the cursor drawing channel. After writing is complete, the cursor drawing module performs a pointer swap, making the BackFramebuffer the new FrontFramebuffer, while the original FrontFramebuffer becomes the BackFramebuffer. Subsequently, when the first moment arrives, the pixel data in the new FrontFramebuffer is read and output to the screen, thus making the cursor appear on the user interface at the predicted position.

[0136] The above embodiments, based on maintaining a cursor drawing channel independent of the system's native drawing process outside the operating system's graphics compositor, monitor the display's screen refresh cycle and generate a software vertical synchronization signal synchronized with the refresh cycle. Furthermore, in response to this software vertical synchronization signal, cursor drawing and frame buffer writing are performed at a second moment before the first moment. This avoids invalid over-rendering and screen tearing caused by data writing during frame buffer reading. Additionally, by responding to the software vertical synchronization signal, calculating the cursor's predicted position at the first moment based on interactive input data and using a motion trajectory prediction algorithm, and then drawing according to this predicted position, the uneven and discrete remote control input events can be converted into a continuous and uniform display position sequence synchronized with the screen refresh cycle. This eliminates abrupt cursor step size changes and visual jitter caused by input interval fluctuations, avoids using outdated historical positions, and makes the cursor movement perceived by the user more closely resemble the actual control intent, improving the smoothness of remote control interaction. The text discusses various aspects of cursor display, including its smoothness and naturalness. It highlights how combining dynamic screen refresh rate adaptation with cursor trajectory prediction can alleviate cursor lag and jitter, resulting in a smoother and more continuous cursor display and improved overall cursor performance. The text also mentions how, by detecting changes in the screen refresh rate and redetermining the first moment of the next screen refresh based on the current refresh rate, and adjusting the software vertical synchronization signal period accordingly, the cursor drawing channel can automatically adapt to displays with different refresh rates (e.g., 50Hz, 60Hz, 120Hz, 240Hz, 288Hz) and dynamic switching scenarios without manual intervention or service restarts, improving the compatibility and robustness of the solution across different hardware platforms and playback scenarios. Finally, it emphasizes that by using the software vertical synchronization signal as the rhythm to drive cursor drawing, rather than triggering drawing with each remote control input event, one effective drawing can be performed within a screen refresh rate cycle, avoiding multiple invalid drawings caused by remote control input frequencies exceeding the screen refresh rate and preventing over-rendering.

[0137] In an exemplary embodiment, the controller executes a second time step prior to the first time step, responding to a software vertical synchronization signal, and calculates the cursor prediction position at the first time step based on interactive input data using a motion trajectory prediction algorithm. Specifically, this step is configured as follows: Figure 8 The steps shown are as follows:

[0138] Step S801: In response to the software vertical synchronization signal at the second moment, determine whether to execute cursor image drawing; Step S802: If it is determined that cursor image drawing will be executed, extract the historical cursor position of the remote control from the recorded interactive input data; Step S803: Based on the historical cursor position of the remote control, generate the cursor prediction position of the cursor at the first moment through a motion trajectory prediction algorithm.

[0139] In response to the software vertical synchronization signal, when the second moment arrives, the cursor drawing module can make a conditional judgment based on the current system state (e.g., whether the cursor is within the screen display area, whether the user has turned off the cursor through settings, whether the cursor resource has been fully loaded, whether it is in standby or paused interaction state, etc.) to determine whether the cursor image needs to be drawn in this vertical synchronization cycle. If the judgment result is not to execute the cursor image drawing, all subsequent drawing steps can be skipped, and the system can wait for the next software vertical synchronization signal; if the judgment result is to execute the cursor image drawing, the system can enter the prediction and drawing process.

[0140] The cursor drawing module can internally maintain interactive input data received from the remote control. This interactive input data can at least include a timestamp and the corresponding cursor coordinates. The cursor history position is a series of coordinate points and their corresponding timestamps extracted from the recorded interactive input data, representing the cursor's movement over a past period on the screen. The cursor history position reflects information such as the cursor's trajectory, speed, and acceleration, and serves as the input basis for the motion trajectory prediction algorithm.

[0141] In the prediction process, the cursor drawing module can input the extracted historical cursor position into the motion trajectory prediction algorithm. The motion trajectory prediction algorithm can perform extrapolation or filtering calculations based on the historical cursor position and output the cursor prediction position at the first moment.

[0142] The above embodiment, after responding to the software vertical synchronization signal at the second moment, first determines whether to execute cursor image drawing (e.g., checking whether the cursor is within the screen display area, whether the user has closed the cursor, whether the cursor resources have been loaded, etc.). This allows subsequent prediction, drawing, and frame buffer writing operations to be skipped directly when the cursor does not need to be displayed, avoiding unnecessary calculations and memory accesses, and reducing processing load and system power consumption. If it is determined that cursor image drawing should be executed, the historical cursor position of the remote control is extracted from the recorded interactive input data. This provides a continuous and stable historical trajectory sequence for the motion trajectory prediction algorithm, enabling the motion trajectory prediction algorithm to make predictions based on the actual movement trend of the cursor (speed, acceleration, etc.), thereby improving the accuracy and reliability of the cursor prediction position.

[0143] In one exemplary embodiment, the controller performs the step of generating the cursor's predicted position at the first moment based on the cursor's historical position from the remote controller using a motion trajectory prediction algorithm, specifically configured as follows: Figure 9 The steps shown are as follows:

[0144] Step S901: Identify the cursor historical position with a later timestamp from the cursor historical position of the remote control; Step S902: Perform cursor position prediction processing based on the cursor historical position with a later timestamp to generate the cursor prediction position of the cursor at the first moment.

[0145] The cursor history position of the remote control is a set of data recorded internally by the controller. Each data entry contains at least a timestamp (recording the time the position was reported) and the corresponding cursor coordinates on the screen. The cursor history position can be stored in chronological order to reflect the cursor's movement trajectory over a past period of time.

[0146] Cursor history positions with later timestamps are those whose timestamps are closest to the current moment among a number of cursor history positions. These typically correspond to the most recent input events reported by the remote control. For example, if the log saves the most recent 10 cursor history positions, the three with the largest timestamps can be considered as later cursor history positions. Compared to cursor history positions with earlier timestamps, later timestamp cursor history positions better reflect the current movement trend of the cursor.

[0147] The cursor drawing module can input the selected cursor historical position with a later timestamp into the motion trajectory prediction algorithm. The motion trajectory prediction algorithm can predict the cursor position based on the cursor historical position with a later timestamp, and thus output the cursor prediction position at the first moment.

[0148] This embodiment identifies the cursor's historical position with a later timestamp from the cursor's historical position in the remote control. This allows the data that best reflects the cursor's most recent movement to be prioritized when performing cursor position prediction processing, avoiding smoothing or misleading by outdated data, and making the predicted cursor position closer to the actual cursor movement.

[0149] In one exemplary embodiment, the controller performs the step of determining whether to draw the cursor image, specifically configured to... Figure 10 The steps shown are as follows:

[0150] Step S1001: Determine whether the current cursor position of the remote control is within the screen of the display; Step S1002: If the current cursor position of the remote control is within the screen of the display, determine whether the cursor display status of the remote control is on; Step S1003: If the cursor display status of the remote control is on, determine to execute cursor image drawing.

[0151] The cursor drawing module can parse a cursor position from the most recent interactive input data and take that position as the current cursor position. The module can then determine if the current cursor position is within the effective display area of ​​the screen. If the current cursor position is not within the effective display area, cursor image drawing will not be performed, thus avoiding drawing invalid cursor images outside the screen. If the current cursor position is within the effective display area, the module can determine if the cursor display state is on. If the cursor display state is off, it means the user or system does not allow cursor display, and cursor image drawing will not be performed. If the cursor display state is on, it means the user or system allows cursor display, and cursor image drawing will be performed.

[0152] This embodiment first determines whether the current cursor position of the remote control is within the display screen. This allows the drawing process to be terminated in advance if the current cursor position is not within the screen display range, avoiding subsequent prediction calculations, image drawing, and frame buffer writing operations, reducing unnecessary load, and saving processing resources and power consumption. After confirming that the current cursor position is within the screen, it determines whether the cursor display status is on. This allows cursor drawing to be prohibited when the user or system actively turns off the cursor display (e.g., pressing the shortcut key to hide the cursor, entering do-not-disturb mode, or a specific application requiring the cursor to be hidden). This achieves accurate response to user interaction preferences and enhances the customizability and scene adaptability of the system.

[0153] In one exemplary embodiment, after receiving interactive input data from the remote controller, the controller is further configured to record the interactive input data from the remote controller in real time.

[0154] Each time the cursor drawing module receives interactive input data from the remote control, it can store the interactive input data in an internally maintained history queue. The interactive input data can at least include the movement data related to the cursor coordinates reported by the remote control and its corresponding timestamp.

[0155] For example, the cursor drawing module monitors the wireless communication link of the remote control (e.g., Bluetooth or infrared). Whenever it receives new interactive input data (including cursor coordinates and timestamp), it stores the interactive input data in chronological order into an internally maintained history queue. This history queue can be, but is not limited to, a fixed-length circular buffer. When the amount of stored data exceeds a preset window size (e.g., retaining a maximum of the most recent 20 inputs), the oldest interactive input data can be automatically overwritten, thus ensuring that the timing information of cursor movement within the most recent period can be obtained at any time.

[0156] This embodiment records the interactive input data of the remote control in real time. Therefore, the cursor's historical position within a recent period can be directly extracted from the record each time the software vertical synchronization signal is triggered. This provides a continuous, complete, and ordered input sequence for the motion trajectory prediction algorithm, ensuring the feasibility and accuracy of the prediction results.

[0157] In one exemplary embodiment, the controller performs the generation of a software vertical synchronization signal synchronized with the screen refresh cycle, specifically configured as follows:

[0158] Determine the software vertical synchronization signal period that is synchronized with the screen refresh cycle; the software vertical synchronization signal generation time corresponding to the software vertical synchronization signal period is before the second time; generate the software vertical synchronization signal at the software vertical synchronization signal generation time according to the software vertical synchronization signal period.

[0159] The software vertical synchronization signal period is the time interval used by the vertical synchronization signal simulated in software. The cursor drawing module determines the software vertical synchronization signal period, which is equal to the monitored screen refresh rate (for example, when the screen refresh rate is 60Hz, the software vertical synchronization signal period is 16.67ms), so that the vertical synchronization signal generated by the software is synchronized with the hardware vertical synchronization signal.

[0160] The software vertical synchronization signal generation time refers to the predetermined point in time on the timeline, according to the software vertical synchronization signal cycle, used to actually generate the software vertical synchronization signal. Setting the software vertical synchronization signal generation time before the second moment (i.e., before the cursor position prediction and cursor drawing operations) allows sufficient processing time, thus achieving precise synchronization between the cursor and screen refresh. The cursor drawing module can generate the software vertical synchronization signal according to the software vertical synchronization signal cycle, at the software vertical synchronization signal generation time within each cycle. This software vertical synchronization signal will serve as the trigger for subsequent cursor prediction and drawing operations. It should also be noted that setting the vertical synchronization signal generation time before the second moment is because it takes time for the generated software vertical synchronization signal to be transmitted to the controller and for the controller to respond; moreover, this setting ensures that the generated software vertical synchronization signal can be successfully responded to by the controller at the second moment.

[0161] In this embodiment, the software vertical synchronization signal is generated before the second moment, thus reserving a fixed lead time (e.g., 2 to 5 milliseconds) for subsequent operations such as cursor prediction, trajectory calculation, image drawing, and frame buffer writing within each screen refresh cycle. This avoids the cursor drawing task failing to complete within the current screen refresh cycle due to insufficient processing time, ensuring that cursor updates are displayed on time. By using a software vertical synchronization signal period equal to the screen refresh cycle and generating the software vertical synchronization signal at the time of its generation according to the software vertical synchronization signal period, the triggering rhythm of the cursor drawing module can be kept consistent with the screen refresh rhythm, avoiding uneven rendering, frame skipping, or screen tearing caused by frequency deviation or phase misalignment between the two.

[0162] In one exemplary embodiment, before determining the first moment of the next screen refresh based on the current screen refresh rate of the display upon detecting a change in the display's screen refresh cycle, the controller is further configured to... Figure 11 The steps shown are as follows:

[0163] Step S1101: Real-time acquisition of monitoring information on screen refresh cycle changes for the display; Step S1102: Based on the monitoring information on screen refresh cycle changes, determine whether the screen refresh cycle of the display has changed.

[0164] Screen refresh rate change monitoring information is notification data obtained through a pre-registered monitoring interface or active query mechanism, indicating whether the screen refresh rate has changed and the new cycle value. This monitoring information can serve as a basis for sensing changes in the screen refresh rate.

[0165] The cursor drawing module registers a screen refresh rate change listener with the operating system's graphics compositor. When the underlying display driver detects a screen refresh rate switch (e.g., due to a change in video format, user settings adjustment, or activation of dynamic screen refresh rate functionality), the operating system's graphics compositor, by calling back this listener, can transmit the new screen refresh rate value as screen refresh rate change monitoring information to the cursor drawing module in real time. Upon receiving the screen refresh rate change monitoring information, the cursor drawing module obtains the new screen refresh rate value and compares it with the currently internally recorded screen refresh rate. If the two values ​​are inconsistent, it is determined that the screen refresh rate has changed; if they are consistent, it is determined that no change has occurred.

[0166] In this embodiment, by acquiring real-time monitoring information on changes in the screen refresh rate of the display, the change can be captured in a timely manner when the screen refresh rate changes (e.g., due to video format switching or user setting adjustments), avoiding perception lag and ensuring that cursor drawing can respond promptly to adjustments in the screen refresh rate.

[0167] In one exemplary embodiment, such as Figure 12 As shown, the operating system graphics compositor may include a vsync management unit, a refresh cycle change callback interface (which can be referred to as the IVsyncPeriodChangedListener interface), layers, and a hardware compositor. As previously described, the cursor drawing module may include a cursor management unit and a signal simulation unit; the signal simulation unit may also be called a screen refresh vertical synchronization signal software simulation unit. Additionally, the cursor drawing module may also include a trajectory prediction unit, a cursor drawing unit native input reader, and a cursor resource management unit, such as... Figure 12 As shown.

[0168] The cursor management unit can interact with the vsync management unit in the operating system's graphics compositor. Layers are the basic units managed by the operating system's graphics compositor; each application window, system status bar, navigation bar, or cursor layer typically corresponds to one layer. Each layer has an independent graphics buffer and compositing parameters such as position, size, transparency, and Z-order. During the compositing phase, the hardware compositor combines the graphics buffers of all layers from bottom to top according to the Z-order of the layers into a complete image frame. Simultaneously, the hardware compositor can also receive hardware vsync signals and pass them to the vsync management unit to drive the timing of layer compositing and buffer exchange.

[0169] The controller executes the step of determining the first moment of the next screen refresh based on the current screen refresh rate when a change in the display's screen refresh cycle is detected. Specifically, this step is configured as follows: Figure 13 The steps shown are as follows:

[0170] Step S1301: When the operating system graphics synthesizer detects a change in the display's screen refresh cycle, it sends the current screen refresh cycle of the display to the cursor management unit. Step S1302: The cursor management unit sends the current screen refresh cycle to the signal simulation unit. Step S1303: The signal simulation unit determines the first moment of the next screen refresh of the display based on the current screen refresh cycle and sends the generated software vertical synchronization signal to the cursor management unit, so that the cursor management unit receives the software vertical synchronization signal at a second moment before the first moment.

[0171] To obtain real-time updates to the frame buffer refresh rate, a custom refresh cycle change callback interface based on AIDL can be defined. This interface uses the operating system's graphics compositor as the server and the cursor drawing module as the client. When the operating system's graphics compositor detects a change in the hardware vsync refresh cycle (e.g., the screen refresh rate switches from 60Hz to 50Hz or 120Hz), it actively calls this interface to send the new screen refresh cycle to the cursor drawing module. Specifically, the cursor management unit implements the client for the refresh cycle change callback interface, and the cursor management unit can also register a vsync cycle change listener with the operating system's graphics compositor. The operating system's graphics compositor adds a refresh cycle change callback interface, which includes the onVsyncPeriodChanged method (refresh cycle change notification method), with the parameter vsync_period (vsync signal period, i.e., the interval between two vsync signals). The vsync management unit in the operating system's graphics compositor detects a change in the interval between two vsync signals (i.e., a change in the screen refresh cycle) through a vsync period change listener. When the vsync management unit initiates a period change callback to the cursor management unit via the refresh cycle change callback interface, specifically, the vsync management unit can call the onVsyncPeriodChanged method in the refresh cycle change callback interface and pass in the latest vsync_period parameter. The cursor management unit can then obtain the new screen refresh cycle. Figure 14 As shown.

[0172] When the cursor management unit receives the `onVsyncPeriodChanged` callback method, it can send the new screen refresh cycle (e.g., switching from 60Hz to 50Hz) carried in the callback parameters to the signal simulation unit. The signal simulation unit updates the software-simulated `vsync` signal cycle in real time and reconfigures the timer of the software vertical synchronization signal through the `timerfd` mechanism to achieve synchronization with the screen refresh rate. Using this timer, before the first moment of the next screen refresh corresponding to the new screen refresh cycle arrives, the signal simulation unit generates the software vertical synchronization signal and sends it to the cursor management unit. Furthermore, the software vertical synchronization signal can be generated before the second moment, so the cursor management unit receives the software vertical synchronization signal at the second moment before the first moment. The `timerfd` (Timer FileDescriptor) mechanism is a timer file descriptor mechanism.

[0173] In this embodiment, the operating system's graphics compositor actively sends the new cycle to the cursor management unit when the screen refresh cycle changes. This establishes a real-time notification path from the underlying refresh rate change to the cursor drawing module, preventing rendering anomalies caused by cycle mismatch in the cursor service. Furthermore, the cursor management unit forwards the screen refresh cycle to the signal simulation unit, ensuring accurate transmission of the screen refresh cycle within the module and guaranteeing the correct parameter source for subsequent synchronization signals. Additionally, the signal simulation unit determines the first moment based on the current screen refresh cycle and generates a software vertical synchronization signal that precedes the first moment. This provides sufficient processing time for cursor prediction and drawing, ensuring that cursor updates are completed before the start of the next screen refresh. Upon receiving the software vertical synchronization signal at the second moment, the cursor management unit can trigger prediction and drawing on time, ensuring that the cursor rendering frequency remains synchronized with the screen refresh rate and eliminating jitter, tearing, and unresponsiveness issues caused by screen refresh rate changes.

[0174] In an exemplary embodiment, the controller executes a second moment prior to the first moment, responding to a software vertical synchronization signal, calculating the cursor's predicted position at the first moment based on interactive input data using a motion trajectory prediction algorithm, drawing a target cursor image from the remote control based on the predicted cursor position, and directly drawing it to the display's frame buffer via the cursor drawing channel. This step is specifically configured as follows: Figure 15 The steps shown are as follows:

[0175] Step S1501: The cursor management unit responds to the received software vertical synchronization signal at the second moment, determines to execute cursor image drawing, and sends a trajectory prediction command to the trajectory prediction unit; Step S1502: The trajectory prediction unit responds to the trajectory prediction command, and generates the cursor prediction position at the first moment based on the historical cursor position of the remote control in the recorded interactive input data through a motion trajectory prediction algorithm; Step S1503: The cursor management unit obtains the cursor prediction position from the trajectory prediction unit, and based on the cursor prediction position, calls the cursor drawing unit to perform cursor drawing processing, generates the target cursor image of the remote control, and draws it directly to the frame buffer of the display through the cursor drawing channel.

[0176] After receiving the software vertical synchronization signal, the cursor management unit can first determine whether to execute cursor drawing. When it is determined to execute cursor drawing, the cursor management unit can send a trajectory prediction command to the trajectory prediction unit. The trajectory prediction command is a control signal or call command sent by the cursor management unit to the trajectory prediction unit, which is used to trigger the trajectory prediction unit to execute a motion trajectory prediction algorithm based on the cursor's historical position to calculate the predicted position of the cursor at a future time (e.g., the first time mentioned above).

[0177] In response to the trajectory prediction command, the trajectory prediction unit can extract the cursor history position of the remote control from the interactive input data recorded in the storage unit, obtain the cursor history position with the later timestamp, run the motion trajectory prediction algorithm based on the cursor history position with the later timestamp, calculate the cursor prediction position of the cursor at the first moment, and return the cursor prediction position to the cursor management unit.

[0178] The cursor management unit calls the cursor drawing unit based on the predicted cursor position. The cursor drawing unit directly writes the target cursor image into the frame buffer according to the predicted cursor position to complete the cursor display on the screen. Specifically, based on an independent cursor drawing channel, the cursor drawing unit can write the pixel data of the target cursor image into the address area corresponding to the predicted cursor position in the Back Framebuffer. After writing is complete, the cursor management unit can perform pointer swapping, making the Back Framebuffer that was just written become the new Front Framebuffer, while the original Front Framebuffer becomes the Back Framebuffer. Subsequently, when the first moment arrives, the pixel data in the new Front Framebuffer is read and output to the screen, thus making the cursor appear on the user interface at the predicted position.

[0179] In this embodiment, the cursor management unit responds to the software vertical synchronization signal at the second moment, first determining whether to execute drawing. This allows the process to be terminated early when the cursor does not need to be displayed, avoiding invalid prediction and drawing operations and reducing system load. The trajectory prediction unit responds to the trajectory prediction command and generates the predicted position at the first moment based on the cursor's historical position. This decouples prediction calculation from drawing, allowing the motion trajectory prediction algorithm to focus on motion trend analysis and improve prediction accuracy. In addition, after obtaining the predicted position, the cursor management unit calls the cursor drawing unit to directly write it to the frame buffer, ensuring the tight timing from prediction to rendering. This allows the drawing result to be written before the first moment, eliminating jitter, tearing, and unresponsiveness caused by input fluctuations, and improving the smoothness of remote control interaction.

[0180] The solution provided in this application introduces the vsync signal into the cursor drawing channel. The generation cycle of the software vsync signal is consistent with the screen refresh cycle (e.g., both are 16.67ms), thus ensuring that the cursor drawing rhythm is consistent with the screen refresh rate. When each software vsync signal is triggered, the cursor management unit selects the cursor position with the timestamp closest to the trigger time of the software vsync signal from the received remote control interaction input data, draws it, and performs pointer swapping after writing it to the BackFramebuffer. The controller reads the FrontFramebuffer in each screen refresh cycle (t1, t2, t3…) and displays the cursor at the corresponding position. Figure 16 The example shown illustrates the advantages of introducing the vsync signal into the cursor drawing channel:

[0181] Assume the arrival times of the remote control's interactive input data p1, p2, p3, and p4 are uneven. If the arrival time of p3 is close to the trigger time of the software vsync signal 2, then when the software vsync signal is triggered, p3 is used for drawing, instead of waiting for p2 or p4. Because the delay between drawing and display is compressed to approximately one screen refresh cycle, and each drawing uses the latest valid input data, the cursor can respond to user operations more promptly, improving responsiveness.

[0182] Figure 16 In the displayed results (Image 0, Image 1, Image 3, Image 5, Image 6), Image 1 corresponds to p1, Image 3 corresponds to p3, Image 5 corresponds to p5, and Image 6 corresponds to p6. P2 and P4, which were not selected because their arrival times were far from the software vsync signal, were skipped. This indicates that the system no longer blindly renders every input, but instead selects representative cursor positions at a fixed pace. This avoids the jitter and frame drops caused by input overlay in traditional schemes. Furthermore, the strategy of using the closest interactive input data ensures the real-time display of the cursor.

[0183] When the software vsync signal 4 is triggered, if no new remote control input data has been received (e.g., p7 has not yet arrived), the cursor management unit writes the most recently recorded input data (e.g., p6) into the BackFramebuffer and performs pointer swapping, resulting in the cursor position displayed on the screen at time t5 being p6. However, the user's actual pointing position may already be close to the expected p7, causing display lag. To address this, this embodiment introduces a trajectory prediction strategy in the cursor drawing channel, such as... Figure 17 As shown; based on the trajectory prediction strategy, even when interactive input data is missing or delayed, the cursor position can be predicted based on the cursor's historical position to obtain the predicted cursor position (e.g., predicting the position of p7), and then drawn based on this predicted cursor position. This makes the cursor display closer to the user's current actual pointing, improving responsiveness. Figure 17 As shown, when the cursor management unit receives the software vsync signal 1, it can obtain the predicted cursor position (denoted as p1') from the trajectory prediction unit for the current screen refresh cycle, and write the target cursor image to the Back Framebuffer and perform pointer swapping according to this predicted cursor position. Subsequently, at the next screen refresh time t2, the contents of the Front Framebuffer are read, and the cursor displayed on the screen is located at the predicted cursor position p1'. In this way, the delay from the triggering of the software vsync signal to the actual display of the cursor is stably controlled within one screen refresh cycle (i.e., one vsync cycle). It can be understood that... Figure 17 t1, t2, t3, etc. in the above correspond to the first time point mentioned above, the time of responding to software vsync signals such as software vsync signal 1, software vsync signal 2, and software vsync signal 3 corresponds to the second time point mentioned above; the time of generating the software vsync signal is before the second time point.

[0184] The cursor drawing channel process provided in this application, involving vsync signals and trajectory prediction strategies, is as follows: Figure 18As shown. Step S1801: Start the cursor drawing thread, create a remote control data reading sub-thread and a screen refresh synchronization signal sub-thread, and register a vsync period change listener with the operating system's graphics compositor. Step S1802: Initialize cursor resources and frame buffer drawing resources in the main thread, and create each unit in the cursor drawing module. Step S1803: The remote control data reading sub-thread reads the interactive input data reported by the remote control, parses it, and injects the interactive input data into the trajectory prediction unit. Step S1804: The screen refresh sub-thread obtains the current hardware vsync signal generation period (which can be denoted as vsync_period), sets the software vsync signal generation period according to the hardware vsync signal generation period, and calls the cursor management unit; wherein, the set software vsync signal generation period is equal to the hardware vsync signal generation period.

[0185] In steps S1805 and S1806, each time the software vsync signal is triggered, the cursor management unit determines whether the cursor is within the effective display area of ​​the screen based on the cursor's most recent historical position and whether the user needs to display the cursor (i.e., whether the cursor display status is on). If both conditions are met, the cursor is determined to meet the display requirements; if either condition is not met, the subsequent drawing steps are skipped, and the system continues to wait for the next software vsync signal.

[0186] Step S1807: When it is determined that the cursor meets the display requirements, the cursor management unit obtains the predicted cursor position from the trajectory prediction unit based on the cursor's historical position. Step S1808: Based on the predicted cursor position, the cursor management unit triggers the cursor drawing unit to directly draw the target cursor image to the frame buffer.

[0187] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise expressly stated herein, there is no strict order restriction on the execution of these steps, and they may be executed in other orders.

[0188] This application also provides a cursor display method, the method including... Figure 7 The steps shown indicate that the method can be applied to a display device, the display device including a display and a controller; the display is configured to: display a user interface; and independently display a cursor on the user interface, the position of the cursor on the user interface being determined by the position pointed to by a remote control in three-dimensional space; the controller is coupled to the display and maintains a cursor drawing channel independent of the system's native drawing process, outside of the operating system graphics synthesizer of the display.

[0189] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the various method embodiments described above.

[0190] In one embodiment, a computer program product is provided having a computer program stored thereon, the computer program being executed by a processor of the steps described in the various method embodiments above.

[0191] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0192] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this application. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A display device, characterized in that, include: The display is configured to show the user interface; Furthermore, a cursor is displayed independently on the user interface, and the position of the cursor on the user interface is determined by the position pointed to by the remote control in three-dimensional space; A controller, coupled to the display and operating outside the display's operating system graphics synthesizer, maintains a cursor drawing channel independent of the system's native drawing process; the controller is configured to: Receive interactive input data from the remote control; Monitor the screen refresh cycle of the display and generate a software vertical synchronization signal that is synchronized with the screen refresh cycle; If a change in the screen refresh cycle of the display is detected, the first moment of the next screen refresh of the display is determined based on the current screen refresh cycle of the display. At a second moment before the first moment, in response to the software vertical synchronization signal, the cursor's predicted position at the first moment is calculated based on the interactive input data using a motion trajectory prediction algorithm. The target cursor image of the remote control is drawn based on the cursor prediction position and drawn directly to the frame buffer of the display via the cursor drawing channel, so that the cursor is drawn on the user interface at the first moment and displayed at the cursor prediction position.

2. The display device according to claim 1, characterized in that, The step of the controller executing the second moment before the first moment, in response to the software vertical synchronization signal, and calculating the cursor prediction position at the first moment based on the interactive input data using a motion trajectory prediction algorithm, is specifically configured as follows: In response to the software vertical synchronization signal at the second moment, it is determined whether to execute cursor image drawing; If it is determined that a cursor image drawing will be performed, the cursor history position of the remote control is extracted from the recorded interactive input data; Based on the cursor's historical position on the remote control, the predicted cursor position at the first moment is generated using a motion trajectory prediction algorithm.

3. The display device according to claim 2, characterized in that, The step of the controller performing the step of generating the predicted cursor position at the first moment based on the cursor's historical position from the remote controller using a motion trajectory prediction algorithm is specifically configured as follows: Identify the cursor history position with the latest timestamp from the cursor history position of the remote control; Based on the cursor's historical position with a later timestamp, cursor position prediction processing is performed to generate the cursor's predicted position at the first moment.

4. The display device according to claim 2, characterized in that, The controller is specifically configured to perform the step of determining whether to draw the cursor image as follows: Determine whether the current cursor position of the remote control is within the screen of the display; If the current cursor position of the remote control is within the screen of the display, determine whether the cursor display state of the remote control is on. If the cursor display on the remote control is enabled, then the cursor image drawing will be executed.

5. The display device according to claim 2, characterized in that, After receiving interactive input data from the remote controller, the controller is further configured to: The system records the interactive input data of the remote control in real time.

6. The display device according to claim 1, characterized in that, The controller executes the generation of a software vertical synchronization signal synchronized with the screen refresh cycle, specifically configured as follows: Determine the software vertical synchronization signal period that is synchronized with the screen refresh cycle; the software vertical synchronization signal generation time corresponding to the software vertical synchronization signal period is before the second time. The software vertical synchronization signal is generated at the time of its generation, according to the software vertical synchronization signal period.

7. The display device according to claim 1, characterized in that, Before determining the first moment of the next screen refresh based on the current screen refresh cycle of the display, upon detecting a change in the display's screen refresh cycle, the controller is further configured to: Real-time acquisition of monitoring information regarding changes in the screen refresh cycle of the display; Based on the screen refresh cycle change monitoring information, it is determined whether the screen refresh cycle of the display has changed.

8. The display device according to claim 1, characterized in that, The display device further includes a cursor drawing module coupled to the controller, the cursor drawing module including a cursor management unit and a signal simulation unit; The controller's step of determining the first moment of the next screen refresh based on the current screen refresh cycle of the display when a change in the display's screen refresh cycle is detected is specifically configured as follows: When the operating system's graphics synthesizer detects a change in the display's screen refresh cycle, it sends the current screen refresh cycle of the display to the cursor management unit. The cursor management unit is controlled to send the current screen refresh cycle to the signal simulation unit; The control signal simulation unit determines the first moment of the next screen refresh of the display based on the current screen refresh cycle, and sends the generated software vertical synchronization signal to the cursor management unit, so that the cursor management unit receives the software vertical synchronization signal at a second moment before the first moment.

9. The display device according to claim 8, characterized in that, The cursor drawing module further includes a trajectory prediction unit and a cursor drawing unit; The controller executes a second time before the first time, responding to the software vertical synchronization signal, and calculates the predicted cursor position at the first time based on the interactive input data using a motion trajectory prediction algorithm. Based on the predicted cursor position, it draws the target cursor image of the remote control and directly draws it to the frame buffer of the display via the cursor drawing channel. Specifically, this step is configured as follows: The cursor management unit is controlled to respond to the received software vertical synchronization signal at the second moment, determine to execute cursor image drawing, and send trajectory prediction instructions to the trajectory prediction unit; The trajectory prediction unit is controlled to respond to the trajectory prediction command and, based on the recorded cursor history position of the remote control in the interactive input data, generates the cursor prediction position of the cursor at the first moment through a motion trajectory prediction algorithm. The cursor management unit is controlled to obtain the predicted cursor position from the trajectory prediction unit, and based on the predicted cursor position, the cursor drawing unit is called to perform cursor drawing processing to generate the target cursor image of the remote control, which is then directly drawn to the frame buffer of the display via the cursor drawing channel.

10. A cursor display method, characterized in that, The invention is applied to a display device, which includes a display and a controller; the display is configured to: display a user interface; and independently display a cursor on the user interface, the position of which is determined by the position pointed to by a remote controller in three-dimensional space; the controller is coupled to the display and maintains a cursor drawing channel independent of the system's native drawing process, outside of the display's operating system graphics synthesizer. The method includes: Receive interactive input data from the remote control; Monitor the screen refresh cycle of the display and generate a software vertical synchronization signal that is synchronized with the screen refresh cycle; If a change in the screen refresh cycle of the display is detected, the first moment of the next screen refresh of the display is determined based on the current screen refresh cycle of the display. At a second moment before the first moment, in response to the software vertical synchronization signal, the cursor's predicted position at the first moment is calculated based on the interactive input data using a motion trajectory prediction algorithm. The target cursor image of the remote control is drawn based on the cursor prediction position and drawn directly to the frame buffer of the display via the cursor drawing channel, so that the cursor is drawn on the user interface at the first moment and displayed at the cursor prediction position.

Images