Display device and interaction method

Abstract

The invention discloses a display device and an interaction method, and the method comprises the steps: receiving an infrared signal from a remote controller, generating a key instruction based on the infrared signal, and executing the key instruction; under the condition that the key instruction is a focus moving instruction, the execution result of the key instruction comprises that a focus frame displayed on the user interface is switched from a current control to a target control; and controlling a mouse cursor on the user interface to move from the current control to the target control according to the coordinates of the current control and the coordinates of the target control.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and an interaction method. Background Art

[0002] With the rapid development of display devices, the functions they can provide users are becoming increasingly rich. Display devices can be controlled through a variety of input devices, such as remote controls and mice. Traditionally, users control display devices using input devices, which have a relatively simple response method and a poor user experience. Summary of the Invention

[0003] The present application provides a display device and an interaction method to solve the problem that the display device has a single response mode when a user uses an input device to control the display device.

[0004] In a first aspect, some embodiments provide a display device, including:

[0005] A display configured to: display a user interface, the user interface including at least one control;

[0006] The processor is configured to:

[0007] receiving an infrared signal from a remote control, generating a key command based on the infrared signal, and executing the key command; in the case where the key command is a focus movement command, the execution result of the key command includes switching a focus frame displayed on the user interface from the current control to the target control;

[0008] Controls the movement of the mouse cursor on the user interface from the current control to the target control according to the coordinates of the current control and the target control.

[0009] When the user presses the direction key on the remote control, not only will the focus frame switch, but the mouse cursor will also move from the original focused control to the later focused control, allowing the user to clearly see the focus change trajectory, improving the user operation experience.

[0010] In some embodiments, the processor is further configured to:

[0011] Check whether the coordinates of the target control and the coordinates of the control that gained focus after the previous focus move overlap. If so, no processing is performed. If not, the mouse cursor on the user interface is moved from the current control to the target control based on the coordinates of the current control and the target control, and the coordinates of the control that gained focus after the previous focus move are updated to the coordinates of the target control. This avoids wasting processing resources by controlling the movement of the mouse cursor even when coordinates overlap.

[0012] In some embodiments, the processor controls the mouse cursor on the user interface to move from the current control to the target control based on the coordinates of the current control and the coordinates of the target control, and is further configured to:

[0013] Based on the coordinates of the current control and the target control, the straight-line distance between the two controls is calculated and used as the trajectory of the virtual mouse movement. The mouse cursor on the user interface is controlled to move along this trajectory, accelerating first and then decelerating within a preset time. This ensures smooth movement.

[0014] In some embodiments, the processor controls the mouse cursor on the user interface to move from the current control to the target control based on the coordinates of the current control and the coordinates of the target control, and is further configured to:

[0015] Determine the display device model; based on the model, determine the upper and lower limits of the coordinates within which the mouse cursor can move; if the coordinates of the target control exceed these limits, modify the coordinates of the target control based on the limits to obtain modified coordinates; and control the movement of the mouse cursor on the user interface based on these modified coordinates. This prevents the mouse cursor from moving outside the visible area of ​​the display device.

[0016] In a second aspect, some embodiments provide a display device, including:

[0017] A display configured to: display a user interface, the user interface including at least one control;

[0018] The processor is configured to:

[0019] receiving mouse movement data from a mouse device, and controlling movement of a mouse cursor on a user interface based on the mouse movement data;

[0020] Generate a control interaction event based on the mouse movement data. When the control interaction event is moving into the target control, switch the focus frame to the target control.

[0021] When a user uses a mouse device to control a display device, not only the mouse cursor moves, but the focus frame also switches accordingly, achieving dual response of the mouse cursor and focus frame, and improving the user operation experience.

[0022] In a third aspect, some embodiments provide an interaction method applied to a display device, wherein the display device includes a driver, a button module, an application performance management (APM), a browser engine, Odin, and a browser interface; the method includes:

[0023] The driver receives infrared signals from the remote control and generates key commands based on the infrared signals. The key commands are transmitted to the browser interface through the key module, APM, browser engine and Odin in sequence.

[0024] The browser interface executes the key command; if the key command is a focus movement command, the execution result of the key command includes the focus frame displayed on the user interface switching from the current control to the target control, and the browser interface transmits the coordinates of the current control and the coordinates of the target control to the browser engine;

[0025] The browser engine controls the mouse cursor on the user interface to move from the current control to the target control according to the coordinates of the current control and the coordinates of the target control.

[0026] When the user presses the direction key on the remote control, not only will the focus frame switch, but the mouse cursor will also move from the original focused control to the later focused control, allowing the user to clearly see the focus change trajectory, improving the user operation experience.

[0027] In some embodiments, the method further comprises:

[0028] The driver receives mouse movement data from the mouse device, generates mouse movement instructions based on the mouse movement data, and transmits the mouse movement instructions to the browser engine through APM; the browser engine controls the movement of the mouse cursor on the user interface based on the mouse movement data, generates control interaction events based on the mouse movement data, and sends the control interaction events to the browser interface through Odin; when the control interaction event is moving into the target control, the browser interface switches the focus box to the target control.

[0029] When a user uses a mouse device to control a display device, not only the mouse cursor moves, but the focus frame also switches accordingly, achieving dual response of the mouse cursor and focus frame, and improving the user operation experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0032] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0033] Figure 3 A schematic diagram of the hardware configuration of a control device provided in some embodiments of the present application;

[0034] Figure 4 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0035] Figure 5 Processing logic flow of the processor provided in some embodiments of the present application Figure 1 ;

[0036] Figure 6 Schematic diagram of a focus frame and a mouse cursor provided for some embodiments of the present application;

[0037] Figure 7 Processing logic flow of the processor provided in some embodiments of the present application Figure 2 ;

[0038] Figure 8 Module interaction process in a display device provided in some embodiments of the present application Figure 1 ;

[0039] Figure 9 Module interaction process in a display device provided in some embodiments of the present application Figure 2 ;

[0040] Figure 10 Module interaction process in a display device provided in some embodiments of the present application Figure 3 . DETAILED DESCRIPTION

[0041] The following embodiments are described in detail, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following embodiments are not intended to represent all possible implementations consistent with the present application. They are merely examples of systems and methods consistent with certain aspects of the present application, as detailed in the claims.

[0042] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0043] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," etc. are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or sequence, unless otherwise noted. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances.

[0044] The terms "comprise," "include," and "have," and any variations thereof, are intended to cover but not exclude inclusion; for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0045] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functionality associated with that element.

[0046] In the embodiments of the present application, the display device 200 generally refers to a device capable of displaying images and processing data. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.

[0047] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG, a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0048] The mobile terminal 300 can function as a control device for performing human-computer interaction between a user and the display device 200. The mobile terminal 300 can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, enabling connection and communication via a network communication protocol, enabling one-to-one control operations and data communication. Audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 for synchronized display.

[0049] like Figure 1 As shown in FIG, the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0050] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart TV, Internet Protocol television (IPTV), etc.

[0051] Figure 2 Some embodiments of this application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0052] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0053] In some embodiments, detector 230 is used to collect signals from the external environment or external interactions. For example, detector 230 may include a light receiver, such as a sensor for collecting ambient light intensity; or an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or a sound collector, such as a microphone, for receiving external sounds.

[0054] In some embodiments, the display 260 includes a display component for displaying images and a driver component for driving image display. The display 260 is configured to receive image signals output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces.

[0055] In some embodiments, the communication device 220 is a component used to communicate with an external device or server 400 according to various communication protocol types. The display device 200 can be provided with multiple communication devices 220 depending on the supported communication methods. For example, if the display device 200 supports wireless network communication, the display device 200 can be provided with a communication device 220 including WiFi functionality. If the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including Bluetooth functionality.

[0056] The communication device 220 can establish a communication connection between the display device 200 and an external device or server 400 via a wireless or wired connection. A wired connection can connect the display device 200 to an external device via a data cable, an interface, or other components. A wireless connection can connect the display device 200 to an external device via a wireless signal or wireless network. The display device 200 can establish a connection with an external device directly or indirectly through a gateway, router, or connection device.

[0057] In some embodiments, the controller 250 may include at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processor, and a power processor, and first to nth interfaces for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in a memory. The controller 250 controls the overall operation of the display device 200.

[0058] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0059] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0060] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may further be provided with an external audio output terminal, through which the audio output device may be connected to the display device 200 to output the sound of the display device 200.

[0061] In some embodiments, the user input interface 280 may be configured to receive instructions from a user.

[0062] Figure 3 Some embodiments of this application provide Figure 1 The hardware configuration diagram of the control device in the figure. Figure 3 As shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0063] The control device 100 is configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200 .

[0064] In some embodiments, the control device 100 may be a smart device. For example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0065] In some embodiments, as Figure 1 As shown, the mobile terminal 300 or other intelligent electronic devices can play a similar function as the control device 100 after installing the application for controlling the display device 200 .

[0066] The controller 110 includes a processor 112, RAM 113, ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between internal components and external and internal data processing functions.

[0067] Under the control of the controller 110, the communication interface 130 communicates control signals and data signals with the display device 200. The communication interface 130 may include at least one of a WiFi chip 131, a Bluetooth module 132, an NFC module 133, or other near field communication modules.

[0068] The user input / output interface 140 includes at least one of a microphone 141 , a touch panel 142 , a sensor 143 , a button 144 and other input interfaces.

[0069] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with the communication interface 130, such as a WiFi, Bluetooth, or NFC module, to encode user input commands via the WiFi protocol, Bluetooth protocol, or NFC protocol and transmit them to the display device 200.

[0070] The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0071] The power supply 180 is used to provide operating power support for various components of the control device 100 under the control of the controller.

[0072] To facilitate user interaction, in some embodiments, display device 200 may run an operating system. An operating system is a computer program used to manage and control the hardware and software resources of display device 200. The operating system can provide a user interface (control the display device), allow users to interact with display device 200, and support the running of various application programs.

[0073] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for display devices.

[0074] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 4 As shown, Figure 4 Some embodiments of this application provide Figure 1 Schematic diagram of software configuration in the display device. In some embodiments, the system of the display device 200 can be divided into three layers, namely, application layer, middleware layer and hardware layer from top to bottom.

[0075] The application layer mainly includes commonly used applications on the TV and the application framework. Common applications are mainly browser-based applications, such as HTML5 apps, and native apps.

[0076] The Application Framework is a complete program model that has all the basic functions required by standard application software, such as file access, data exchange, etc., as well as the user interfaces of these functions (toolbars, status bars, menus, dialog boxes).

[0077] Native apps can support online or offline, message push or local resource access.

[0078] The middleware layer includes various television protocols, multimedia protocols, and system components. Middleware uses the basic services (functions) provided by system software to connect various parts of application systems or different applications on the network, enabling resource and function sharing.

[0079] The hardware layer primarily includes the Hardware Abstraction Layer (HAL) interface, hardware, and drivers. The HAL interface serves as a unified interface for all TV chipsets, while the specific logic is implemented by each chip. Drivers primarily include audio drivers, display drivers, Bluetooth drivers, camera drivers, Wi-Fi drivers, USB drivers, HDMI drivers, sensor drivers (such as fingerprint sensors, temperature sensors, and pressure sensors), and power drivers.

[0080] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0081] In some embodiments, a display device is provided, comprising: a display configured to: display a user interface, the user interface including at least one control; the user interface can be a user interface of any page in a browser application, for example, the user interface can be a user interface of a homepage of a browser application, optionally, the control in the embodiment of the present application can also be referred to as an element (DOM); a processor configured to implement the following steps, such as Figure 5 As shown:

[0082] Step 501, receiving an infrared signal from a remote control, generating a key instruction based on the infrared signal, and executing the key instruction; in the case where the key instruction is a focus movement instruction, the execution result of the key instruction includes switching the focus frame displayed on the user interface from the current control to the target control.

[0083] Each button on the remote control corresponds to a key value. When a user presses a button, the remote control sends an infrared signal containing the key value corresponding to the button to the display device.

[0084] After receiving the infrared signal from the remote control, the display device analyzes the infrared signal to obtain the key value therein, and generates a key instruction based on the key value.

[0085] For example, when the key value in the infrared signal is the key value corresponding to the confirmation key, a confirmation key instruction is generated; when the key value in the infrared signal is the key value corresponding to the return key, a page return instruction is generated; when the key value in the infrared signal is the key value corresponding to the up key, a focus up instruction is generated; when the key value in the infrared signal is the key value corresponding to the down key, a focus down instruction is generated; when the key value in the infrared signal is the key value corresponding to the left key, a focus left instruction is generated; when the key value in the infrared signal is the key value corresponding to the right key, a focus right instruction is generated. The above-mentioned focus up instruction, focus down instruction, focus left instruction, and focus right instruction all belong to the focus movement instructions in the embodiments of the present application.

[0086] After generating a key command, the display device can execute the key command. For example, when the key command is a confirmation key command, the execution result of the key command is to jump to the user interface of the page linked to the focused control; when the key command is a page return command, the execution result of the key command is to jump from the user interface of the current page back to the user interface of the previous page; when the key command is a focus move command, the execution result of the key command includes the focus frame displayed on the user interface switching from the current control to the target control.

[0087] Among them, when the key instruction is a focus movement instruction, the process of executing the focus movement instruction includes: releasing the focus of the current focused control, and giving the focus to the target control corresponding to the focus movement instruction. Specifically, the focus attribute of the current focused control can be set to the out-of-focus attribute, and the focus attribute of the target control corresponding to the focus movement instruction can be set to the on-focus attribute. Then, the focus frame displayed on the user interface is switched from the current control to the target control. That is to say, when the key instruction is a focus movement instruction, the execution result of the key instruction includes the focus frame displayed on the user interface being switched from the current control to the target control.

[0088] For example, the currently focused control is control A, the control above control A is control B, the control below control A is control C, the control to the left of control A is control D, and the control to the right of control A is control E. When the key instruction is a focus-up instruction, the process of executing the focus-up instruction includes: setting the focus attribute of control A to the out-of-focus attribute, setting the focus attribute of control B to the top-focus attribute, and switching the focus frame displayed on the user interface from control A to control B; when the key instruction is a focus-down instruction, the process of executing the focus-down instruction includes: setting the focus attribute of control A to the out-of-focus attribute, setting the focus attribute of control C to the top-focus attribute, and switching the focus frame displayed on the user interface from control A to control C; when the key instruction is a focus-left instruction, the process of executing the focus-left instruction includes: setting the focus attribute of control A to the out-of-focus attribute, setting the focus attribute of control D to the top-focus attribute, and switching the focus frame displayed on the user interface from control A to control D; when the key instruction is a focus-right instruction, the process of executing the focus-right instruction includes: setting the focus attribute of control A to the out-of-focus attribute, setting the focus attribute of control E to the top-focus attribute, and switching the focus frame displayed on the user interface from control A to control E.

[0089] Step 502: Control the mouse cursor on the user interface to move from the current control to the target control according to the coordinates of the current control and the coordinates of the target control.

[0090] Among them, when the key instruction is a focus movement instruction, after switching the focus frame displayed on the user interface from the current control to the target control, the coordinates of the current control and the coordinates of the target control can be obtained, and according to the coordinates of the current control and the coordinates of the target control, the mouse cursor on the user interface is controlled to move from the current control to the target control.

[0091] Among them, the control is a rectangular area. The coordinates of the control can be expressed using: top (the distance between the top of the control and the top of the page), bottom (the distance between the bottom of the control and the top of the page), left (the distance between the right side of the control and the left side of the page), right (the distance between the right side of the control and the left side of the page), width (the width of the control), height (the height of the control), x (the horizontal coordinate of the center point of the control), and y (the vertical coordinate of the center point of the control).

[0092] For example, the coordinates of the current control or the target control can be:

[0093] {

[0094] top: 565.796875,

[0095] bottom: 620.296875,

[0096] left:: 429.125,

[0097] right: 585.6875,

[0098] width: 156.5625,

[0099] height:54.5,

[0100] x: 429.125,

[0101] y: 565.796875

[0102] }

[0103] Among them, after obtaining the coordinates of the current control and the target control, the coordinates of the center point of the current control can be determined based on the coordinates of the current control, and the coordinates of the center point of the target control can be determined based on the coordinates of the target control, and the mouse cursor can be controlled to move from the center point coordinates of the current control to the center point coordinates of the target control.

[0104] Alternatively, after obtaining the coordinates of the current control and the target control, the current control's corner coordinates can be determined based on the current control's coordinates, and the target control's corner coordinates can be determined based on the target control's coordinates, and the mouse cursor can be controlled to move from the current control's corner coordinates to the target control's corner coordinates. The corner coordinates here can be the upper left corner coordinates, the upper right corner coordinates, the lower left corner coordinates, or the lower right corner coordinates, and are not limited in this embodiment of the application. Figure 6 As an example, through the solution of the embodiment of the present application, when the focus frame switches to a certain control, the mouse cursor will also move to the control.

[0105] In the above embodiment, a display device is provided, comprising: a display configured to display a user interface, the user interface including at least one control; a processor configured to receive an infrared signal from a remote control, generate a key command based on the infrared signal, and execute the key command; if the key command is a focus movement command, the execution result of the key command includes switching a focus frame displayed on the user interface from the current control to a target control; and controlling a mouse cursor on the user interface to move from the current control to the target control based on the coordinates of the current control and the coordinates of the target control. When a user presses a direction key on the remote control, not only does the focus frame switch, but the mouse cursor also moves from the previously focused control to the later focused control, allowing the user to clearly see the focus change trajectory, thereby improving the user experience.

[0106] In some embodiments, the processor is further configured to: determine whether the coordinates of the target control and the coordinates of the control focused after the previous focus movement are repeated; if repeated, no processing is performed; if not repeated, control the mouse cursor on the user interface to move from the current control to the target control based on the coordinates of the current control and the coordinates of the target control, and update the coordinates of the control focused after the previous focus movement to the coordinates of the target control.

[0107] Consider a special scenario: the current focused control is control A, and control A is in the last row of the user interface. Due to the size limitation of the display, control B below control A is not displayed. In this case, if the key value in the infrared signal is the key value corresponding to the down key, then the process of executing the focus down instruction includes: setting the focus attribute of control A to the out-of-focus attribute, setting the focus attribute of control B to the up-focus attribute, and scrolling the row where control B is located up to appear in the user interface, and switching the focus frame displayed on the user interface from control A to control B; in this case, control B will scroll up to the original position of control A, that is, although the focus has changed, the coordinates of the focused control before the focus change and the coordinates of the focused control after the focus change have not changed. Therefore, there is no need to control the movement of the mouse cursor on the user interface, which can greatly save processing resources.

[0108] Specifically, after obtaining the coordinates of the current control and the target control, it can be determined whether the coordinates of the target control and the coordinates of the control that was focused after the previous focus movement are repeated. If they are repeated, it is considered to be the special scenario mentioned above, and no processing is performed to save processing resources; if they are not repeated, the mouse cursor on the user interface is controlled to move from the current control to the target control based on the coordinates of the current control and the coordinates of the target control, and the coordinates of the control that was focused after the previous focus movement are updated to the coordinates of the target control.

[0109] In the above embodiment, the coordinates of the target control and the coordinates of the control that was focused after the previous focus movement are checked to see if they overlap. If so, no processing is performed. If not, the mouse cursor on the user interface is controlled to move from the current control to the target control based on the coordinates of the current control and the coordinates of the target control, and the coordinates of the control that was focused after the previous focus movement are updated to the coordinates of the target control. This avoids the waste of processing resources caused by controlling the movement of the mouse cursor even when the coordinates overlap.

[0110] In some embodiments, the processor is further configured to: mount a focus change listening program during the initialization phase of the target application where the user interface is located; and obtain the coordinates of the current control and the coordinates of the target control through the focus change listening program when the key instruction is a focus movement instruction.

[0111] The target application where the user interface is located may be a browser application. During the initialization phase of the browser application, a focus change monitoring program may be mounted. The mounted focus change monitoring program is used to monitor focus changes.

[0112] As described above, when the key command is a focus movement command, the process of executing the focus movement command includes: releasing the focus of the currently focused control and giving the focus to the target control corresponding to the focus movement command. After this step is completed, the focus change listener will monitor the focus change. At this time, the browser API (when the target application where the user interface is located is a browser application) can be used, such as the getBoundingClientRect method, to further obtain the coordinates of the focused control before the focus change (current control) and the coordinates of the focused control after the focus change (target control). Then, based on these two coordinates, the mouse cursor on the user interface is controlled to move from the current control to the target control.

[0113] In the above embodiment, a focus change listener is installed during the initialization phase of the target application containing the user interface. When a key press is a focus move command, the focus change listener retrieves the coordinates of the current control and the target control. These two coordinates are then used to control the movement of the mouse cursor from the current control to the target control. This achieves a combined response of the focus frame and the mouse cursor.

[0114] In some embodiments, the processor controls the mouse cursor on the user interface to move from the current control to the target control based on the coordinates of the current control and the coordinates of the target control, and is further configured to: calculate the straight-line distance between the current control and the target control based on the coordinates of the current control and the coordinates of the target control, and use the straight-line distance as the trajectory of the virtual mouse movement; control the mouse cursor on the user interface to complete the movement within a preset time period according to the trajectory, using a method of first accelerating and then decelerating.

[0115] Among them, after obtaining the coordinates of the current control and the target control, the coordinates of the center point of the current control can be determined based on the coordinates of the current control, and the coordinates of the center point of the target control can be determined based on the coordinates of the target control. The straight-line distance between the coordinates of the center point of the current control and the coordinates of the center point of the target control is calculated, and the straight-line distance is used as the trajectory of the virtual mouse movement; the mouse cursor on the user interface is controlled to move according to the trajectory, and the movement is completed within 50ms by first accelerating and then decelerating to maintain the smoothness of the movement process.

[0116] Alternatively, after obtaining the coordinates of the current control and the target control, the coordinates of the lower right corner of the current control can be determined based on the coordinates of the current control, and the coordinates of the lower right corner of the target control can be determined based on the coordinates of the target control. The straight-line distance between the coordinates of the lower right corner of the current control and the coordinates of the lower right corner of the target control is calculated, and the straight-line distance is used as the trajectory of the virtual mouse movement; the mouse cursor on the user interface is controlled to move along the trajectory, using a method of first accelerating and then decelerating to complete the movement within 50ms to maintain the smoothness of the movement process.

[0117] In the above embodiment, the straight-line distance between the current control and the target control is calculated based on the coordinates of the current control and the target control, and the straight-line distance is used as the trajectory of the virtual mouse movement; the mouse cursor on the user interface is controlled to move according to the trajectory, and the movement is completed within the preset time by first accelerating and then decelerating, which can ensure the smoothness of the movement process.

[0118] In some embodiments, the processor controls the mouse cursor on the user interface to move from the current control to the target control based on the coordinates of the current control and the coordinates of the target control, and is further configured to: determine the model of the display device; determine the upper and lower limits of the coordinates that the mouse cursor can move based on the model; when the coordinates of the target control exceed the upper and lower limits of the coordinates, modify the coordinates of the target control based on the upper and lower limits of the coordinates to obtain modified coordinates; and control the movement of the mouse cursor on the user interface based on the modified coordinates.

[0119] Among them, after obtaining the coordinates of the current control and the coordinates of the target control, the model of the display device can be determined first, and the upper and lower limits of the coordinates that the mouse cursor can move can be determined according to the model.

[0120] Optionally, the model of the display device may be, for example, a 4K model or a 2K model. The upper and lower limits of the coordinates include upper and lower limits of the horizontal axis and upper and lower limits of the vertical axis.

[0121] For example, when the display device is a 4K model, it can be determined that the upper limit of the horizontal coordinate that the mouse cursor can move is 1920, the upper limit of the vertical coordinate is 1080, the lower limit of the horizontal coordinate is 0, and the lower limit of the vertical coordinate is 0.

[0122] Then, the coordinates of the center point of the target control can be extracted from the coordinates of the target control, and the horizontal coordinate in the coordinates of the center point of the target control can be compared with the upper limit and lower limit of the horizontal coordinate to obtain a first comparison result; the vertical coordinate in the coordinates of the center point of the target control can be compared with the upper limit and lower limit of the vertical coordinate to obtain a second comparison result; if the first comparison result indicates that the horizontal coordinate is between the corresponding upper and lower limits, and the second comparison result indicates that the vertical coordinate is also between the corresponding upper and lower limits, then it is considered that the coordinates of the target control do not exceed the upper and lower limits of the coordinates; if the first comparison result indicates that the horizontal coordinate is not between the corresponding upper and lower limits, or the second comparison result indicates that the vertical coordinate is not between the corresponding upper and lower limits, then it is considered that the coordinates of the target control exceed the upper and lower limits of the coordinates.

[0123] Among them, when the coordinates of the target control exceed the upper and lower limits of the coordinates, the coordinates of the target control are modified based on the upper and lower limits of the coordinates to obtain the modified coordinates. For example, if the first comparison result indicates that the horizontal coordinate is not between the corresponding upper and lower limits, then it is determined whether the horizontal coordinate is less than the lower limit of the horizontal coordinate or greater than the upper limit of the horizontal coordinate. If it is less than the lower limit of the horizontal coordinate, the horizontal coordinate is modified to the lower limit of the horizontal coordinate; if it is greater than the upper limit of the horizontal coordinate, the horizontal coordinate is modified to the upper limit of the horizontal coordinate; similarly; if the second comparison result indicates that the vertical coordinate is not between the corresponding upper and lower limits, then it is determined whether the vertical coordinate is less than the lower limit of the vertical coordinate or greater than the upper limit of the vertical coordinate. If it is less than the lower limit of the vertical coordinate, the vertical coordinate is modified to the lower limit of the vertical coordinate; if it is greater than the upper limit of the vertical coordinate, the vertical coordinate is modified to the upper limit of the vertical coordinate.

[0124] After the modified coordinates are obtained, the mouse cursor on the control user interface moves from the current position to the modified coordinates.

[0125] In the above embodiment, the model of the display device is determined; the upper and lower coordinate limits of the mouse cursor are determined based on the model; when the coordinates of the target control exceed the upper and lower coordinate limits, the coordinates of the target control are modified based on the upper and lower coordinate limits to obtain modified coordinates; and the movement of the mouse cursor on the user interface is controlled based on the modified coordinates. This can prevent the mouse cursor from moving outside the visible area of ​​the display device.

[0126] In some embodiments, a display device is provided, comprising: a display configured to: display a user interface, the user interface including at least one control; a processor configured to implement the following steps, such as Figure 7 As shown:

[0127] Step 701: Receive mouse movement data from a mouse device, and control the movement of a mouse cursor on a user interface based on the mouse movement data.

[0128] Optionally, the mouse device can be connected to the display device via a USB interface.

[0129] The user can freely switch between the remote control and the mouse device. That is, after the display device is turned on, the user can control the display device at any time using the remote control or the mouse device. When the user uses the mouse device to control the display device, the mouse device sends mouse movement data to the display device. After receiving the mouse movement data, the display device directly controls the movement of the mouse cursor on the user interface based on the mouse movement data, similar to the principle of a computer and a mouse.

[0130] Step 702: Generate a control interaction event based on the mouse movement data. If the control interaction event is a movement into a target control, switch the focus frame to the target control.

[0131] Among them, the display device controls the movement of the mouse cursor on the user interface based on the mouse movement data, and can also generate control interaction events based on the mouse movement data. The control interaction events include, for example, click events, move-in events, move-out events, etc.

[0132] When the display device generates a move-in event of moving into a target control based on the mouse movement data, the focus frame is switched to the target control.

[0133] In the above embodiment, a display device is provided, comprising: a display configured to display a user interface, the user interface including at least one control; and a processor configured to receive mouse movement data from a mouse device, control the movement of a mouse cursor on the user interface based on the mouse movement data, generate a control interaction event based on the mouse movement data, and, if the control interaction event is a move into a target control, switch a focus frame to the target control. When a user uses the mouse device to control the display device, not only does the mouse cursor move, but the focus frame also switches accordingly, achieving dual responsiveness of the mouse cursor and the focus frame, thereby improving the user experience.

[0134] In some embodiments, the target application where the user interface is located is a browser application, and an interaction method is provided, which is applied to a display device, and the display device includes a driver, a button module, an application performance management (Application Performance Management, referred to as APM), a browser engine webengine, Odin and a browser interface WebUI; see Figure 8As shown, the method includes: driving to receive infrared signals from a remote control and generating key instructions based on the infrared signals; transmitting the key instructions to the browser interface through the key module, APM, browser engine and Odin in sequence; the browser interface executes the key instructions; in the case where the key instructions are focus movement instructions, the execution result of the key instructions includes the focus frame displayed on the user interface switching from the current control to the target control, the browser interface transmitting the coordinates of the current control and the coordinates of the target control to the browser engine; the browser engine controls the mouse cursor on the user interface to move from the current control to the target control according to the coordinates of the current control and the coordinates of the target control.

[0135] Among them, the button module and application performance management APM can constitute a DFB module.

[0136] Among them, when the key instruction is a focus movement instruction, the browser interface can first obtain the coordinates of the current focused control (i.e., the current control) from Odin, and then execute the key instruction. The execution process of the key instruction includes releasing the focus of the current focused control and giving the focus to the target control corresponding to the focus movement instruction. After this step is completed, the browser interface obtains the coordinates of the current focused control (i.e., the target control) from Odin again, and then switches the focus frame from the current control to the target control, and then transmits the coordinates of the current control and the coordinates of the target control to the browser engine. The browser engine controls the mouse cursor on the user interface to move from the current control to the target control based on the coordinates of the current control and the coordinates of the target control.

[0137] In the above embodiment, the driver receives an infrared signal from a remote control and generates a key command based on the infrared signal; the key command is transmitted to the browser interface through the key module, APM, browser engine, and Odin in sequence; the browser interface executes the key command; if the key command is a focus movement command, the execution result of the key command includes the focus frame displayed on the user interface switching from the current control to the target control, and the browser interface transmitting the coordinates of the current control and the coordinates of the target control to the browser engine; the browser engine controls the mouse cursor on the user interface to move from the current control to the target control based on the coordinates of the current control and the coordinates of the target control. When the user presses the direction key on the remote control, not only will the focus frame switch, but the mouse cursor will also move from the originally focused control to the later focused control, allowing the user to clearly see the focus change trajectory, thereby improving the user operation experience.

[0138] In some embodiments, the method also includes: the browser engine mounts an OMI message listener during the initialization phase, and the browser interface mounts a focus change listener during the initialization phase; in the case where the key instruction is a focus movement instruction, after the browser interface executes the key instruction, the coordinates of the current control and the coordinates of the target control are obtained through the focus change listener; the browser interface sends an OMI message to the browser engine, and the OMI message includes the coordinates of the current control and the coordinates of the target control; the browser engine obtains the OMI message through the OMI message listener, and obtains the coordinates of the current control and the coordinates of the target control from the OMI message.

[0139] See also Figure 9 As shown, the target application where the user interface is located is a browser application, which includes a browser engine and a browser interface; the browser engine mounts an OMI message listener during the initialization phase, and the browser interface mounts a focus change listener and an OMI message listener during the initialization phase. The driver receives infrared signals from the remote control and generates key commands based on the infrared signals. The key commands are transmitted to the browser interface through the key module, APM, browser engine, and Odin in sequence. The browser interface executes the key commands. If the key command is a focus move command, the execution process of the key command includes releasing the focus of the currently focused control and giving the focus to the target control corresponding to the focus move command. After this step is completed, the focus change listener will monitor the focus change. At this time, the browser API (if the target application of the user interface is a browser application) can be used, such as the getBoundingClientRect method, to further obtain the coordinates of the focused control before the focus change (current control) and the coordinates of the focused control after the focus change (target control). After the focus frame is switched, an OMI message is sent to the browser engine. The OMI message includes the coordinates of the current control and the coordinates of the target control. The browser engine receives the OMI message through the OMI message listener and obtains the coordinates of the current control and the target control from the OMI message. Based on the coordinates of the current control and the target control, the mouse cursor on the user interface is controlled to move from the current control to the target control.

[0140] In the above embodiment, the browser engine mounts an OMI message listener during the initialization phase, and the browser interface mounts a focus change listener during the initialization phase. When the key command is a focus movement command, the browser interface executes the key command and then obtains the coordinates of the current control and the target control through the focus change listener. The browser interface sends an OMI message to the browser engine, and the OMI message includes the coordinates of the current control and the target control. The browser engine obtains the OMI message through the OMI message listener and obtains the coordinates of the current control and the target control from the OMI message. Subsequently, based on the coordinates of the current control and the target control, the mouse cursor on the user interface is controlled to move from the current control to the target control, thus achieving dual response of the focus frame and the mouse cursor.

[0141] In some embodiments, the method further includes: adding the coordinates of the current control into a predefined pre-switching coordinate field to obtain pre-switching coordinate data; adding the coordinates of the target control into a predefined post-switching coordinate field to obtain post-switching coordinate data; and encapsulating the pre-switching coordinate data, the post-switching coordinate data, the predefined message communication type, and the predefined cursor movement mode in a preset format to obtain an OMI message.

[0142] Optionally, the message communication format is as follows:

[0143] {

[0144] type: WEBUI_MOVE_MOUSE,

[0145] oldFocus:{x:0,y:0},

[0146] newsFocus:{x:0,y:0}

[0147] mouseType: 'normal'

[0148] }

[0149] In this format, the pre-switching coordinate field is defined as oldFocus:{x:0,y:0}, the post-switching coordinate field is defined as newsFocus:{x:0,y:0}, the message communication type is defined as type: WEBUI_MOVE_MOUSE, and the cursor movement mode is defined as mouseType: 'normal'. After obtaining the coordinates of the current and target controls, the current control's coordinates are added to oldFocus:{x:0,y:0} to obtain the pre-switching coordinate data; the target control's coordinates are added to newsFocus:{x:0,y:0} to obtain the post-switching coordinate data. After these additions and processing, an OMI message is generated.

[0150] In the above embodiment, the coordinates of the current control are added to a pre-defined pre-switching coordinate field to obtain pre-switching coordinate data; the coordinates of the target control are added to a pre-defined post-switching coordinate field to obtain post-switching coordinate data; the pre-switching coordinate data, post-switching coordinate data, a pre-defined message communication type, and a pre-defined cursor movement mode are packaged according to a preset format to obtain an OMI message. The browser interface then sends this OMI message to the browser engine. The browser engine obtains this OMI message through an OMI message listener, parses it to obtain the coordinates of the current control and the coordinates of the target control, and controls the mouse cursor on the user interface to move from the current control to the target control based on the coordinates of the current control and the target control, achieving dual response of the focus frame and the mouse cursor.

[0151] In some embodiments, the method also includes: driving to receive mouse movement data from a mouse device, generating mouse movement instructions based on the mouse movement data, and transmitting the mouse movement instructions to the browser engine through APM; the browser engine controls the movement of the mouse cursor on the user interface based on the mouse movement data, generates control interaction events based on the mouse movement data, and sends the control interaction events to the browser interface through Odin; when the control interaction event is to move into the target control, the browser interface switches the focus box to the target control.

[0152] See also Figure 10 As shown, when the user switches to using a mouse device to control the display device, the mouse device will send mouse movement data to the display device. After the display device driver receives the mouse movement data from the mouse device, it generates a mouse movement instruction based on the mouse movement data, and transmits the mouse movement instruction to the browser engine through APM; the browser engine controls the movement of the mouse cursor on the user interface based on the mouse movement data, which is similar to the principle of a computer and a mouse. While the browser engine controls the movement of the mouse cursor on the user interface based on the mouse movement data, it can also generate control interaction events based on the mouse movement data. Control interaction events include, for example, click events, move-in events, move-out events, etc., and send the control interaction events to the browser interface through Odin. When the control interaction event is to move into the target control, the browser interface updates the properties of the currently focused control to the out-of-focus property, updates the properties of the target control to the on-focus property, and switches the focus frame to the target control. This achieves dual response of the mouse cursor and the focus frame.

[0153] In the above embodiment, when the user uses the mouse device to control the display device, not only the mouse cursor moves, but the focus frame also switches accordingly, achieving dual response of the mouse cursor and the focus frame, and improving the user operation experience.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

[0155] For ease of explanation, the above description has been presented in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations are possible. The above embodiments have been selected and described to better explain the principles and practical applications, thereby enabling those skilled in the art to better utilize the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that: include: A display configured to: display a user interface, wherein the user interface includes at least one control; The processor is configured to: receiving an infrared signal from a remote control, generating a key instruction based on the infrared signal, and executing the key instruction; if the key instruction is a focus movement instruction, the execution result of the key instruction includes switching a focus frame displayed on the user interface from a current control to a target control; According to the coordinates of the current control and the coordinates of the target control, the mouse cursor on the user interface is controlled to move from the current control to the target control.

2. The display device according to claim 1, wherein The processor is further configured to: Whether the coordinates of the target control and the coordinates of the control focused after the previous focus movement are repeated; If repeated, no processing will be done; If not repeated, the mouse cursor on the user interface is controlled to move from the current control to the target control according to the coordinates of the current control and the target control, and the coordinates of the control focused after the previous focus movement are updated to the coordinates of the target control.

3. The display device according to claim 1, wherein The processor is further configured to: During the initialization phase of the target application where the user interface is located, mounting a focus change monitoring program; In the case where the key instruction is a focus moving instruction, the coordinates of the current control and the coordinates of the target control are obtained through the focus change monitoring program.

4. The display device according to claim 1, wherein The processor controls the mouse cursor on the user interface to move from the current control to the target control according to the coordinates of the current control and the coordinates of the target control, and is further configured to: Calculate the straight-line distance between the current control and the target control according to the coordinates of the current control and the target control, and use the straight-line distance as the trajectory of the virtual mouse movement; The mouse cursor on the user interface is controlled to move along the trajectory within a preset time period by first accelerating and then decelerating.

5. The display device according to claim 1, wherein The processor controls the mouse cursor on the user interface to move from the current control to the target control according to the coordinates of the current control and the coordinates of the target control, and is further configured to: Determining the model of the display device; Determine the upper and lower limits of the coordinates within which the mouse cursor can move according to the device model; When the coordinates of the target control exceed the upper and lower limits of the coordinates, modifying the coordinates of the target control based on the upper and lower limits of the coordinates to obtain modified coordinates; According to the modified coordinates, the mouse cursor on the user interface is controlled to move.

6. A display device, characterized in that: include: A display configured to: display a user interface, wherein the user interface includes at least one control; The processor is configured to: receiving mouse movement data from a mouse device, and controlling movement of a mouse cursor on the user interface based on the mouse movement data; A control interaction event is generated based on the mouse movement data, and when the control interaction event is a movement into a target control, the focus frame is switched to the target control.

7. An interactive method, characterized in that: Applied to a display device, the display device includes a driver, a button module, an application performance management (APM), a browser engine, Odin, and a browser interface; the method includes: The driver receives an infrared signal from a remote control, generates a key command based on the infrared signal, and transmits the key command to the browser interface through the key module, APM, browser engine, and Odin in sequence; The browser interface executes the key instruction; if the key instruction is a focus movement instruction, the execution result of the key instruction includes switching the focus frame displayed on the user interface from the current control to the target control, and the browser interface transmits the coordinates of the current control and the coordinates of the target control to the browser engine; The browser engine controls the mouse cursor on the user interface to move from the current control to the target control according to the coordinates of the current control and the coordinates of the target control.

8. The method according to claim 7, characterized in that The method further comprises: The browser engine mounts the OMI message listening program during the initialization phase, and the browser interface mounts the focus change listening program during the initialization phase; In the case where the key instruction is a focus movement instruction, after the browser interface executes the key instruction, the coordinates of the current control and the coordinates of the target control are obtained through the focus change monitoring program; the browser interface sends an OMI message to the browser engine, and the OMI message includes the coordinates of the current control and the coordinates of the target control; The browser engine obtains the OMI message through the OMI message listening program, and obtains the coordinates of the current control and the coordinates of the target control from the OMI message.

9. The method according to claim 8, characterized in that The method further comprises: Add the coordinates of the current control to the pre-defined coordinate field before switching to obtain the coordinate data before switching; Add the coordinates of the target control to the predefined post-switching coordinate field to obtain the post-switching coordinate data; The coordinate data before switching, the coordinate data after switching, the predefined message communication type, and the predefined cursor movement mode are encapsulated according to a preset format to obtain the OMI message.

10. The method according to claim 7, characterized in that The method further comprises: The driver receives mouse movement data from the mouse device, generates a mouse movement instruction based on the mouse movement data, and transmits the mouse movement instruction to the browser engine through the APM; The browser engine controls the movement of a mouse cursor on the user interface based on the mouse movement data, generates a control interaction event based on the mouse movement data, and sends the control interaction event to the browser interface through the Odin; When the control interaction event is moving into a target control, the browser interface switches the focus frame to the target control.

Images