Display device and terminal device

Abstract

The invention is applicable to the technical field of display equipment, and provides display equipment and terminal equipment, and the display equipment comprises a display which is configured to display a user interface; the controller is configured to receive touch information sent by the terminal equipment; the terminal equipment is configured with a touch control area, the touch control information comprises a touch control type to which a touch control operation belongs and first touch control position information associated with the touch control operation, and the first touch control position information is used for representing a touch control position associated with the touch control operation in the touch control area; the first touch position information is converted into second touch position information used for representing the position in the user interface; and generating a touch event according to the second touch position information and the touch type, and distributing the touch event to an application currently displayed in the user interface to trigger the application to respond to the touch event. The accuracy of remotely controlling the display equipment by a user can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of display devices, and in particular to a display device and a terminal device. Background Art

[0002] With the improvement of display device performance, display devices (such as smart TVs, car screens, etc.), in addition to traditional playback functions, can also provide streaming services such as online video, music, games, learning, etc. through different applications installed by users, greatly enriching the user experience.

[0003] Currently, remote device control and voice control are commonly used to better control display devices, enabling remote interaction between users and display devices. However, voice control relies heavily on the accuracy of speech recognition, and remote device control can be inaccurate when conveying user actions. Consequently, these control methods can lead to inaccurate remote control when users interact with display devices remotely. Summary of the Invention

[0004] The present application provides a display device and a terminal device, which can improve the accuracy of a user's remote control of the display device.

[0005] In a first aspect, some embodiments of the present application provide a display device, including a display and a controller.

[0006] The display is configured to display a user interface;

[0007] The controller is configured as:

[0008] Receiving touch information sent by a terminal device, wherein the terminal device is configured with a touch area, the touch information including: a touch type to which a touch operation belongs, and first touch position information associated with the touch operation, the first touch position information being used to represent a touch position associated with the touch operation within the touch area;

[0009] converting the first touch position information into second touch position information for representing a position within the user interface;

[0010] A touch event is generated according to the second touch position information and the touch type, and the touch event is distributed to an application currently displayed in the user interface to trigger the application to respond to the touch event.

[0011] In an embodiment of the present application, by receiving touch information sent by a terminal device and converting the first touch position information therein into second touch position information for representing the position within the user interface, the touch operation on the terminal device can be effectively mapped to the display device, and then a touch event can be constructed based on the above-mentioned second touch position information and touch type, and the touch operation generated by the user on the terminal device can be converted into a touch event in the display device. Finally, the touch event is distributed to the application currently displayed on the user interface, so that the user can remotely control the application in the terminal device through the touch operation on the terminal device, avoiding the problem of difficulty in controlling the display device when voice recognition is inaccurate, and improving the user's interactive experience and accuracy in remotely controlling the display device.

[0012] In a second aspect, some embodiments of the present application further provide a terminal device, including a display and a controller.

[0013] a display configured to display a touch area;

[0014] The controller is configured as:

[0015] In response to a touch operation input in the touch area, triggering a control event corresponding to the touch operation;

[0016] Determining first touch position information associated with the touch operation;

[0017] Determine the touch type of the touch operation according to the control event, send the first touch position information and the touch type to the display device, so that the display device generates a touch event according to the first touch position information and the touch type, and triggers an application to respond to the touch event.

[0018] In an embodiment of the present application, by detecting a control event corresponding to a touch operation, the first touch position information associated with the above touch operation is determined, and the touch type to which the touch operation belongs is determined based on the control event, the first touch position information and the touch type are sent to the display device, so that the display device can generate a touch event based on the first touch position information and the touch type to simulate the user's operation on the terminal device, and trigger the application to respond to the above touch event.

[0019] In a third aspect, some embodiments of the present application further provide a display method, applied to the display device provided in the first aspect, the method comprising:

[0020] Receiving touch information sent by a terminal device, wherein the terminal device is configured with a touch area, the touch information including: a touch type to which a touch operation belongs, and first touch position information associated with the touch operation, the first touch position information being used to represent a touch position associated with the touch operation within the touch area;

[0021] converting the first touch position information into second touch position information for representing a position within the user interface;

[0022] A touch event is generated according to the second touch position information and the touch type, and the touch event is distributed to an application currently displayed in the user interface to trigger the application to respond to the touch event.

[0023] In a fourth aspect, some embodiments of the present application further provide a display method, applied to the terminal device provided in the second aspect, the method comprising:

[0024] In response to a touch operation input in the touch area, triggering a control event corresponding to the touch operation;

[0025] Determining first touch position information associated with the touch operation;

[0026] Determine the touch type of the touch operation according to the control event, send the first touch position information and the touch type to the display device, so that the display device generates a touch event according to the first touch position information and the touch type, and triggers an application to respond to the touch event.

[0027] In a fifth aspect, a display device is provided, comprising a unit for executing the display method in the third aspect or the fourth aspect.

[0028] In a sixth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a display device, the display device executes the display method in the third aspect or the fourth aspect.

[0029] In a seventh aspect, a computer program product is provided, the computer program product comprising: a computer program, which, when executed by a display device, causes the display device to execute the display method in the third aspect or the fourth aspect.

[0030] It can be understood that the beneficial effects of the third to seventh aspects mentioned above can be found in the relevant descriptions of the first and second aspects mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0032] Figure 1A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

[0033] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;

[0034] Figure 3 A schematic diagram of the hardware configuration of a control device provided in some embodiments of the present application;

[0035] Figure 4 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;

[0036] Figure 5 A structural block diagram of a display device provided in some embodiments of the present application;

[0037] Figure 6 A schematic flow chart of a display method applied to a display device provided in some embodiments of the present application;

[0038] Figure 7 A structural block diagram of a terminal device provided in some embodiments of the present application;

[0039] Figure 8 A schematic flow chart of a display method applied to a terminal device provided in some embodiments of the present application;

[0040] Figure 9 A schematic structural diagram of a display device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0041] The following embodiments are described in detail, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following embodiments are not intended to represent all possible implementations consistent with the present application. They are merely examples of systems and methods consistent with certain aspects of the present application, as detailed in the claims.

[0042] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0043] In the description of the embodiments of this application, unless otherwise specified, " / " represents or. For example, A / B can represent A or B. "And / or" in this article is simply a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of this application, "plurality" means two or more than two.

[0044] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," etc. are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or sequence, unless otherwise noted. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances.

[0045] The terms "comprise," "include," and "have," and any variations thereof, are intended to cover but not exclude inclusion; for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0046] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functionality associated with that element.

[0047] In the embodiments of the present application, the display device 200 generally refers to a device capable of displaying images and processing data. For example, the display device 200 includes, but is not limited to, a car screen, a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, and the like.

[0048] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG, a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.

[0049] The mobile terminal 300 can function as a control device for performing human-computer interaction between a user and the display device 200. The mobile terminal 300 can also function as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, enabling connection and communication via a network communication protocol, enabling one-to-one control operations and data communication. Audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 for synchronized display.

[0050] like Figure 1 As shown in FIG, the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0051] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart TV, Internet Protocol television (IPTV), etc.

[0052] Figure 2 Some embodiments of this application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.

[0053] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0054] In some embodiments, detector 230 is used to collect signals from the external environment or external interactions. For example, detector 230 may include a light receiver, such as a sensor for collecting ambient light intensity; or an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or a sound collector, such as a microphone, for receiving external sounds.

[0055] In some embodiments, the display 260 includes a display component for presenting images and a driver component for driving image display. The display 260 is configured to receive image signals output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces.

[0056] In some embodiments, the display 260 can be a liquid crystal display (LCD), an organic light-emitting diode display (OLED), a plasma display panel (PDP), a vacuum fluorescent display (VFD), a field emission display (FED), or a projection display, and can also be a projection device or a projection screen.

[0057] In some embodiments, the communication device 220 is a component used to communicate with an external device or server 400 according to various communication protocol types. The display device 200 can be provided with multiple communication devices 220 depending on the supported communication methods. For example, if the display device 200 supports wireless network communication, the display device 200 can be provided with a communication device 220 including WiFi functionality. If the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including Bluetooth functionality.

[0058] The communication device 220 can establish a communication connection between the display device 200 and an external device or server 400 via a wireless or wired connection. A wired connection can connect the display device 200 to an external device via a data cable, an interface, or other components. A wireless connection can connect the display device 200 to an external device via a wireless signal or wireless network. The display device 200 can establish a connection with an external device directly or indirectly through a gateway, router, or connection device.

[0059] In some embodiments, the controller 250 may include at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processor, and a power processor, and first to nth interfaces for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in a memory. The controller 250 controls the overall operation of the display device 200.

[0060] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0061] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).

[0062] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may further be provided with an external audio output terminal, through which the audio output device may be connected to the display device 200 to output the sound of the display device 200.

[0063] In some embodiments, the user input interface 280 may be configured to receive instructions from a user.

[0064] Figure 3 Some embodiments of this application provide Figure 1 The hardware configuration diagram of the control device in the figure is as follows. Figure 3 As shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0065] The control device 100 is configured to control the display device 200 , and can receive user input operation instructions, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200 .

[0066] In some embodiments, the control device 100 may be a smart device. For example, the control device 100 may be installed with various applications for controlling the display device 200 according to user needs.

[0067] In some embodiments, as Figure 1 As shown, the mobile terminal 300 or other intelligent electronic devices can play a similar function as the control device 100 after installing the application for controlling the display device 200 .

[0068] The controller 110 includes a processor 112, RAM 113, ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication and cooperation between internal components and external and internal data processing functions.

[0069] Under the control of the controller 110, the communication interface 130 communicates control signals and data signals with the display device 200. The communication interface 130 may include at least one of a WiFi chip 131, a Bluetooth module 132, an NFC module 133, or other near field communication modules.

[0070] The user input / output interface 140 includes at least one of a microphone 141 , a touch panel 142 , a sensor 143 , a button 144 and other input interfaces.

[0071] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with the communication interface 130, such as a WiFi, Bluetooth, or NFC module, to encode user input commands via the WiFi protocol, Bluetooth protocol, or NFC protocol and transmit them to the display device 200.

[0072] The memory 190 is used to store various operating programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can store various control signal instructions input by the user.

[0073] The power supply 180 is used to provide operating power support for each component of the control device 100 under the control of the controller.

[0074] To facilitate user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling the hardware and software resources in the display device 200. The operating system may provide a user interface (control the display device), allow the user to interact with the display device 200, and support the running of various application programs.

[0075] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform. In the embodiment of the present application, the specific operating platform may be a mobile platform, and the operating system may be a mobile operating system based on the mobile platform, such as Android, iOS, HarmonyOS, or a third-party operating system customized and developed based on the Android system.

[0076] The operating system can be divided into different modules or layers according to the functions implemented, e.g. Figure 4 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (referred to as "application layer"), the application framework layer (referred to as "framework layer"), the system library layer and the kernel layer.

[0077] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with users based on these applications. The application layer can host at least one application, which can include a system setup program or clock program included with the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0078] The framework layer provides applications with an application programming interface (API) and programming framework. The application framework layer includes predefined functions. The application framework layer acts as a processing center, determining the actions taken by applications in the application layer. Through the API, applications can access system resources and services during execution.

[0079] like Figure 4As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to the system location service; a package manager for retrieving various information related to the application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0080] In some embodiments, the activity manager is used to manage the life cycle of each application and common navigation back functions, such as controlling the exit, opening, and back of the application.

[0081] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library or the Android runtime library, to implement the functions to be implemented by the framework layer.

[0082] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 4 As shown, the kernel layer can be configured with hardware drivers, and the drivers included in the kernel layer can be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0083] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.

[0084] In some related technologies, as the functions of display devices are improved, in addition to the traditional remote control method of remote controllers, display devices can also provide a variety of remote control methods to realize remote interaction between users and display devices, including: voice control, gesture control, remote device control and other remote control methods.

[0085] However, in the above-mentioned remote control methods, voice control and gesture control are limited by the performance of the display device, and voice control is heavily dependent on the accuracy of semantic recognition, and gesture control is also heavily dependent on the accuracy of gesture recognition; remote device control is generally achieved through an analog remote control, that is, by displaying analog remote control buttons on the remote device (mobile phone, tablet, etc.), and the user achieves remote control by clicking these analog buttons. However, the user needs to click these analog buttons frequently, and due to the limitations of the analog button functions, the user operations cannot be accurately mapped to the display device. Therefore, the above-mentioned remote control method will lead to inaccurate remote control of the user.

[0086] In order to improve the accuracy of user remote control of a display device, an embodiment of the present application provides a display device and a terminal device, in which the display device receives touch information sent by the terminal device; the above-mentioned touch information includes: the touch type to which the touch operation belongs, and first touch position information associated with the touch operation, and the above-mentioned first touch position information is used to characterize the touch position associated with the above-mentioned touch operation in the touch area; the first touch position information is converted into second touch position information for characterizing the position in the user interface; a touch event is generated according to the second touch position information and the touch type, and the above-mentioned touch event is distributed to the application currently displayed in the user interface to trigger the application to respond to the above-mentioned touch event.

[0087] In order to facilitate a further understanding of the technical solutions in some embodiments of the present application, the technical solutions of the display device and the display method, as well as how the technical solutions solve the above-mentioned technical problems, are described in detail below in conjunction with some specific embodiments and drawings. The various embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Obviously, the described embodiments are part of the embodiments of the present application, not all of them.

[0088] Figure 5 Shows some embodiments of the present application Figure 1 For the convenience of explanation, only the parts related to the embodiment of the present application are shown in the structural block diagram of the display device.

[0089] Reference Figure 5 The display device includes a display 510 and a controller 520 , and the controller 520 is connected to the display 510 .

[0090] in,

[0091] The display 510 is configured to display a user interface.

[0092] The controller 520 is configured to:

[0093] Receiving touch information sent by a terminal device, wherein the terminal device is configured with a touch area, and the touch information includes: a touch type to which a touch operation belongs, and first touch position information associated with the touch operation, wherein the first touch position information is used to represent a touch position associated with the touch operation within the touch area;

[0094] Converting the first touch position information into second touch position information for representing a position within the user interface;

[0095] A touch event is generated according to the second touch position information and the touch type, and the touch event is distributed to an application currently displayed in the user interface to trigger the application to respond to the touch event.

[0096] Figure 6 A flow chart of a display method provided in some embodiments of the present application is shown. The display method is applied to a display device and is described in detail as follows:

[0097] S610. Receive touch information sent by the terminal device; the touch information includes: the touch type to which the touch operation belongs, and first touch position information associated with the touch operation, the first touch position information being used to represent the touch position associated with the touch operation within the touch area.

[0098] It should be understood that the aforementioned terminal device is configured with a touch area. When a user performs a touch operation on the touch area, the user interface of the display device will implement the aforementioned touch operation through a corresponding cursor, thereby enabling the user to remotely control the display device. For example, when a user performs a sliding operation on the touch area, the cursor on the display device will also perform a corresponding sliding operation; when a user performs a click operation on the touch area, the cursor on the display device will also perform a corresponding click operation.

[0099] Among them, the above-mentioned touch type may include a terminal touch type or a display touch type. The above-mentioned terminal touch type is used to indicate the type of touch operation responded by the terminal device, and the above-mentioned display touch type is used to indicate the type of response of the display device based on the above-mentioned touch operation; the above-mentioned first touch position information can be the touch area position information, touch coordinates, etc. corresponding to the user's touch operation.

[0100] It should also be understood that when the first touch position information is touch coordinates, the touch coordinates received by the display device may include first touch coordinates or second touch coordinates. The first touch coordinates are the touch point coordinates associated when the touch operation is triggered, and the second touch coordinates are the touch point coordinates adjusted based on the touch trajectory corresponding to the touch operation and the movement of the terminal device when the touch operation is triggered.

[0101] Optionally, when the user performs a touch operation in the touch area, a corresponding control event can be triggered, and the terminal touch type (such as touchstart, touchmove, etc.) can be determined based on the touch event, and then sent to the display device in real time; or, the terminal device can determine the display control type (such as ACTION_DOWN, ACTION_MOVE, etc.) corresponding to the display based on the terminal touch type (such as touchstart, touchmove, etc.) and other touch information (such as touch time, touch pressure, etc.).

[0102] It should be noted that when the touch type is a terminal touch type, in order to improve the accuracy of the display device's touch event encapsulation, the touch information may also include one or more of touch pressure, touch duration, number of touches, and sliding distance. Of course, for the purpose of subsequent functional expansion, when the touch type is a display touch type, the touch information may also include touch pressure, touch duration, number of touches, sliding distance, and other information, which is not limited here.

[0103] S620: Convert the first touch position information into second touch position information for representing a position in the user interface.

[0104] It should be understood that the above-mentioned first touch position information is the touch position associated with the touch operation within the touch area. In order to improve the accuracy of user control of the display device, the first touch position information can be mapped to the above-mentioned user interface according to the preset position correspondence, that is, the first touch position information is converted into second touch position information for representing the position within the above-mentioned user interface.

[0105] It should also be understood that the second touch position information corresponds to a cursor in the display device, that is, the cursor moves based on the second touch position information.

[0106] S630: Generate a touch event according to the second touch position information and the touch type, and distribute the touch event to an application currently displayed in the user interface to trigger the application to respond to the touch event.

[0107] It should be understood that the above touch events are analog events that can be responded to by the display device, including click events, slide events and other operation events, etc. Among them, the above operation events may include short press events, long press events, fast slide events, etc.

[0108] Specifically, the controller can generate a click event and a slide event based on the above-mentioned second touch position information and the above-mentioned touch type, and distribute them to the application currently displayed in the user interface; at the same time, the controller can also determine a short press event or a long press event based on the above-mentioned click event and touch time (for example, a click event is a short press event when the touch time is less than a preset time threshold, and a long press event when the touch time is greater than or equal to the preset time threshold); determine whether it is a fast slide event based on the slide event, touch time and slide distance (for example, a slide event is a fast slide event when the sliding speed determined by the touch time and sliding distance is greater than a preset speed threshold).

[0109] In an embodiment of the present application, by receiving touch information sent by a terminal device and converting the first touch position information therein into second touch position information for representing the position within the user interface, the touch operation on the terminal device can be effectively mapped to the display device, and then a touch event can be constructed based on the above-mentioned second touch position information and touch type, and the touch operation generated by the user on the terminal device can be converted into a touch event in the display device. Finally, the touch event is distributed to the application currently displayed on the user interface, so that the user can remotely control the application in the terminal device through the touch operation on the terminal device, avoiding the problem of difficulty in controlling the display device when voice recognition is inaccurate, and improving the user's interactive experience and accuracy in remotely controlling the display device.

[0110] In some embodiments, before converting the first touch position information into second touch position information representing a position within the user interface, the controller is further configured to:

[0111] receiving first screen information sent by the terminal device, and acquiring second screen information of the display;

[0112] The coordinate scaling ratio is determined according to the first screen information and the second screen information.

[0113] It should be understood that the first screen information may be the screen size of the terminal device, and the second screen information may be the screen size of the display device. Of course, since the display area of ​​the display device or terminal device may not be consistent with the screen size, the first screen information and the second screen information may be the size of the actual display area. For example, the first screen information may be the size of the touch area of ​​the terminal device, and the second screen information may be the size of the user interface of the display device.

[0114] Specifically, the controller can convert position information through the MotionParser class, including obtaining the corresponding first screen information and second screen information according to the system interface (API) of the terminal device and the display device, and calculating the coordinate scaling ratio, wherein the above-mentioned coordinate scaling ratio includes the horizontal scaling ratio and the vertical scaling ratio.

[0115] For example, if the terminal device is a mobile phone and the display device is a TV, the first screen information includes phoneWidth (i.e., mobile phone width) and phoneHeight (i.e., mobile phone height), and the second screen information includes tvWidth (i.e., TV width) and tvHeight (i.e., TV height). Then the horizontal scaling ratio scaleFactorX = tvWidth / phoneWidth, and the vertical scaling ratio is scaleFactorY = tvHeight / phoneHeight.

[0116] Correspondingly, when the controller converts the first touch position information into second touch position information for representing the position in the user interface, the controller is configured to:

[0117] The first touch position information is converted into the second touch position information according to the coordinate scaling ratio.

[0118] Specifically, the horizontal coordinates in the first touch position information are converted according to the horizontal scaling factor to obtain the horizontal coordinates in the second touch position information, and the vertical coordinates in the first touch position information are converted according to the vertical scaling factor to obtain the vertical coordinates in the second touch position information. For example, if the first touch position information is (phoneX, phoneY), then in the second touch position information (tvX, tvY), tvX = phoneX * scaleFactorX, and tvY = phoneY * scaleFactorY.

[0119] In the embodiment of the present application, position information adapted to the display device can be obtained through position information conversion so that the display device can perform corresponding operations or event processing.

[0120] In some embodiments, before the controller generates a touch event according to the second touch position information and the touch type, the controller is further configured to:

[0121] Correcting the second touch position information to obtain corrected touch position information;

[0122] It should be understood that the controller will determine the above-mentioned second touch position information in real time, and the second touch position information is closely related to the cursor movement in the display device and the event processing in the display device. Therefore, the second touch position information can be corrected through the preset signal processing method provided by the pointer acceleration engine (PointerAccelEngine class), thereby improving the accuracy of the second touch position information, making the cursor movement in the display device smoother and the event processing in the display device more accurate.

[0123] Correspondingly, when the controller performs the correction of the second touch position information, it is configured to:

[0124] The second touch position information is corrected by a preset signal processing method, wherein the preset signal processing method includes at least one of the following: signal smoothing processing, signal filtering processing, signal interpolation processing, and signal deletion processing.

[0125] Among them, the above-mentioned signal smoothing processing refers to smoothing the second touch position information (such as touch coordinates) through an algorithm (such as a moving average algorithm) to reduce jitter caused by hand shaking or other factors; the above-mentioned signal filtering processing refers to applying filtering technology (such as a low-pass filter) to remove high-frequency noise in the second touch position information and retain the main trend of the signal; the above-mentioned signal interpolation processing or signal deletion processing refers to adding missing signal points (interpolation) or removing redundant signal points (deletion) according to the timestamp and sampling rate of the second touch position information to ensure the temporal consistency and smoothness of the second touch position information.

[0126] For example, if the second touch position information is touch coordinates, the touch coordinates may be first subjected to signal smoothing processing, then signal filtering processing, and finally signal interpolation processing and signal deletion processing to obtain corrected touch coordinates.

[0127] In the embodiment of the present application, the converted position information can be effectively corrected through a preset signal processing method, ensuring that the touch operation on the display device is smoother and more accurate.

[0128] In some embodiments, when the controller generates a touch event according to the second touch position information and the touch type, the controller is configured to:

[0129] The touch event is generated according to the corrected touch position information and the touch type.

[0130] It should be understood that after obtaining the corrected touch position information, constructing a touch event according to the corrected touch position information and the touch type can refer to the above embodiments, which will not be described in detail here.

[0131] In some embodiments, when the controller constructs the touch event according to the corrected touch position information and the touch type, the controller is configured to:

[0132] Create an input event;

[0133] The event attribute of the input event is set according to the corrected touch position information and the touch type to obtain the touch event.

[0134] It should be understood that the MotionEvent class can be used to simulate the user's touch operation and obtain corresponding touch events.

[0135] Specifically, when using the MotionEvent class to simulate a user's touch operation, first create an input event (such as a MotionEvent) based on the Coordinate object through the event creation method (such as the createMotionEvent method) of the motion event factory class (such as the MotionEventFactory class). Then, the corrected touch position information and the touch type (such as ACTION_DOWN, ACTION_MOVE, ACTION_UP, etc.) are set as event attributes of the input event (such as MotionEvent) to obtain the corresponding touch event. Of course, other event attributes can also be set according to actual conditions, such as touch pressure, touch duration, etc.

[0136] In the embodiment of the present application, the MotionEvent class can be used to convert the user's touch operation on the terminal device into a touch event (such as MotionEvent) that can be recognized by the display device.

[0137] In some embodiments, each application currently displayed in the above user interface is provided with an independent application window.

[0138] It should be understood that each application in the user interface is displayed through an independent window, that is, each application corresponds to an independent window.

[0139] Correspondingly, when the controller executes the distribution of the touch event to the application currently displayed in the user interface, it is configured to:

[0140] determining a target window according to the second touch position information, wherein the target window is an application window displayed on the top layer of the user interface and including the second touch position information;

[0141] Distribute the touch event to the application corresponding to the target window.

[0142] Specifically, the controller can implement the distribution of touch events through the input management class (InputManager). When implementing the distribution of touch events through the input management class (InputManager), the InputManager instance is called through the acquisition method (e.g., InputManager.getInstance() method) in the encapsulation tool class (InputManagerWrapper). During this process, the controller traverses each independent window and then determines that the application window displayed at the top of the user interface and containing the above-mentioned second touch position information is the target window. Then, the created touch event (i.e., MotionEvent object) is distributed to the application corresponding to the target window using the injection method (e.g., injectInputEvent() method). It should be noted that in split-screen mode, that is, when including multiple independent windows, the above-mentioned target window is the application window that is first triggered by the second touch position information. For example, the user interface includes window A (corresponding to application A) and window B (corresponding to application B). When the trajectory composed of the second touch position information is from window A to window B, since the window corresponding to application A is triggered first, all subsequent touch events constructed based on the second touch position information will be distributed to application A.

[0143] In the embodiment of the present application, the touch events can be distributed to corresponding applications through the above-mentioned input management class (InputManager), so that each application can respond to the touch events in a timely manner.

[0144] Figure 7 A structural block diagram of a terminal device provided by some embodiments of the present application is shown. For ease of explanation, only the parts related to the embodiments of the present application are shown.

[0145] Reference Figure 7 The terminal device includes a display 710 and a controller 720 , and the controller 720 is connected to the display 710 .

[0146] in,

[0147] The display 710 is configured to display a touch area.

[0148] The controller 720 is configured to:

[0149] In response to a touch operation input in the touch area, triggering a control event corresponding to the touch operation;

[0150] Determining first touch position information associated with the touch operation;

[0151] Determine the touch type of the touch operation according to the control event, send the first touch position information and the touch type to the display device, so that the display device generates a touch event according to the first touch position information and the touch type, and triggers the application to respond to the touch event.

[0152] Figure 8 A flow chart of a display method provided in some embodiments of the present application is shown. The display method is applied to a terminal device and is described in detail as follows:

[0153] S810: In response to a touch operation input in the touch area, trigger a control event corresponding to the touch operation.

[0154] It should be understood that the terminal device may be a mobile phone, tablet, watch, or other device; the terminal device may be connected to the display device via a network or Bluetooth. The terminal device may be installed with a touch application (APP) for remotely controlling the display device. After the user opens the touch application, a touch area for the user to perform touch operations may be displayed on the display. The touch operations include click operations, slide operations, etc. The control events may be touch events, etc.

[0155] S820: Determine first touch position information associated with the touch operation.

[0156] It should be understood that the first touch position information may be touch area position information or touch coordinates corresponding to the touch operation.

[0157] Specifically, when the first touch position information is the touch coordinates, the touch coordinates associated with the user's touch operation include the first touch coordinates or the second touch coordinates. The controller may use the real-time touch point coordinates corresponding to when the touch operation is triggered as the above-mentioned first touch coordinates. At the same time, since the terminal device may not be stationary when the user performs a touch operation, that is, the movement of the terminal device may affect the user's touch operation, the real-time touch point coordinates may be corrected according to the movement of the terminal device (such as acceleration, distance, movement trajectory, etc.) to obtain the above-mentioned second touch coordinates. Optionally, the real-time touch point coordinates may be adjusted according to the touch trajectory corresponding to the touch operation and the movement of the terminal device to obtain the second touch coordinates.

[0158] In the embodiment of the present application, by determining the first touch coordinates or the second touch coordinates associated with the touch operation, the accuracy of the position associated with the touch operation can be improved.

[0159] S830. Determine the touch type of the touch operation according to the control event, and send the first touch position information and the touch type to the display device, so that the display device generates a touch event according to the first touch position information and the touch type, and triggers the application to respond to the touch event.

[0160] It should be understood that the terminal device and the display device can respond to different event types, so the above touch type can include a terminal touch type or a display touch type. Among them, the terminal touch type is the type of touch operation that the terminal device can respond to, for example, the terminal touch type can be touchstart, touchmove, etc.; the display touch type is the type of touch operation that the display device can respond to, for example, the display touch type can be ACTION_DOWN, ACTION_MOVE, etc.

[0161] Specifically, when a user performs a touch operation in the touch area, a corresponding control event can be triggered, and the terminal touch type is determined based on the touch event, and then sent to the display device in real time. At this time, the display device determines the event type it can respond to. Alternatively, the terminal device can directly determine the display touch type corresponding to the display (such as ACTION_DOWN, ACTION_MOVE, etc.) based on the terminal touch type (such as touchstart, touchmove, etc.) and other touch information (such as touch time, touch pressure, etc.). The determination of the above touch type can be set according to actual conditions and is not limited here.

[0162] In this embodiment of the present application, when the terminal device performance is poor, the terminal touch type can be directly transmitted to reduce the terminal device's computing pressure; when the display device performance is poor, the terminal device can determine the display touch type to reduce the display device's computing pressure. This flexible touch type determination method improves applicability in different scenarios.

[0163] In an embodiment of the present application, by detecting a control event corresponding to a touch operation, the first touch position information associated with the above touch operation is determined, and the touch type to which the touch operation belongs is determined based on the control event, the first touch position information and the touch type are sent to the display device, so that the display device can generate a touch event based on the first touch position information and the touch type to simulate the user's operation on the terminal device, and trigger the application to respond to the above touch event.

[0164] In some embodiments, before the controller determines the first touch position information associated with the touch operation, the controller is further configured to:

[0165] Acquiring movement data of the terminal device, wherein the movement data includes acceleration direction and acceleration magnitude;

[0166] Determine trajectory data of a touch trajectory corresponding to the touch operation, wherein the trajectory data includes a trajectory direction.

[0167] It should be understood that the terminal device may be provided with a sensor, and the sensor may include one or more of a gyroscope, a gravity sensor, a motion sensor, and the like. For example, the magnitude and direction of acceleration of the direction axis (including the three direction axes of x-axis, y-axis, and z-axis) are obtained based on the gyroscope. The touch trajectory corresponding to the touch operation is a trajectory composed of the touch point coordinates corresponding to the touch operation. The direction of the trajectory is the direction from the first touch coordinate corresponding to the touch operation to the touch coordinate currently corresponding to the touch operation, that is, the vector from the first touch coordinate corresponding to the touch operation to the touch coordinate currently corresponding to the touch operation.

[0168] It should also be understood that, depending on actual conditions or subsequent functional expansion, the above movement data may also include movement distance, movement direction, etc.; the above trajectory data may also include trajectory distance, sliding speed, etc., which are not limited here.

[0169] Correspondingly, when the acceleration magnitude is greater than or equal to the preset acceleration threshold, the controller is configured to:

[0170] The first touch position information associated with the touch operation is determined according to the acceleration direction and the trajectory direction.

[0171] Specifically, when the acceleration magnitude is greater than or equal to a preset acceleration threshold, it indicates that the current terminal device is moving too fast, which may affect the user's touch operation. The above-mentioned first touch position information associated with the touch operation can be determined based on the relationship between the acceleration direction and the trajectory direction. When the acceleration direction and the trajectory direction are consistent, the first touch position information associated with the touch operation is adjusted; otherwise, the first touch position information associated with the touch operation is not adjusted.

[0172] For example, when the acceleration in any direction of the x-axis, y-axis, and z-axis is greater than or equal to a preset acceleration threshold, the first touch position information associated with the touch operation is determined based on the relationship between the acceleration direction and the trajectory direction.

[0173] It should be noted that when the acceleration is smaller than the preset acceleration threshold, the impact on the user's touch operation is small, and the touch point coordinates corresponding to the touch operation (i.e., the first touch coordinates when the touch operation is triggered) can be directly used as the above-mentioned first touch position information.

[0174] In some embodiments, when the controller determines the first touch position information associated with the touch operation based on the acceleration direction and the trajectory direction, the controller is configured to:

[0175] In a case where the acceleration direction and the trajectory direction are inconsistent, the first touch coordinates when the touch operation is triggered are determined as the first touch position information associated with the touch operation.

[0176] Specifically, a combined acceleration direction of the acceleration directions can be calculated based on the direction axis. When the difference between the combined acceleration direction and the trajectory direction is greater than or equal to a preset direction threshold, it is determined that the acceleration direction and the trajectory direction are inconsistent. Otherwise, the acceleration direction and the trajectory direction are consistent. In the event that the acceleration direction and the trajectory direction are inconsistent, in order to reduce the impact of the touch operation when the terminal device is moved, the first touch coordinates when the touch operation is triggered are directly used as the first touch position information associated with the above-mentioned touch operation.

[0177] For example, the merged acceleration direction A is calculated based on the acceleration directions of the x-axis and the y-axis. If the trajectory direction is B, when the direction difference between A and B is greater than or equal to the preset direction threshold, it means that the acceleration direction and the trajectory direction are inconsistent. At this time, the coordinates of the trajectory point triggered by the touch operation are used as the first touch position information associated with the above-mentioned touch operation.

[0178] In some embodiments, when the controller determines the first touch position information associated with the touch operation according to the acceleration direction and the trajectory direction, the controller is configured to:

[0179] In a case where the acceleration direction is consistent with the trajectory direction, adjusting the first touch coordinate currently triggered by the touch operation according to the magnitude of the acceleration to obtain a second touch coordinate;

[0180] The second touch coordinates are determined as the first touch position information associated with the touch operation.

[0181] Specifically, a combined acceleration magnitude can be calculated based on the directional axes. When the combined acceleration magnitude is greater than or equal to a preset acceleration threshold, displacement amplification is performed based on the first touch coordinates currently triggered by the touch operation. That is, the preset coordinate value is increased based on the first touch coordinates to obtain second touch coordinates, and the second touch coordinates are used as the first touch position information associated with the touch operation. When the combined acceleration magnitude is less than the preset acceleration threshold, no processing is performed, that is, the first touch coordinates when the touch operation was triggered are directly used as the first touch position information associated with the touch operation.

[0182] It should be understood that the above-mentioned preset coordinate values ​​can be set according to actual conditions and are not limited here.

[0183] For example, the combined acceleration magnitude C is calculated based on the acceleration magnitudes of the x-axis and y-axis. If the trajectory direction is B, when C is greater than or equal to the preset acceleration threshold, the coordinate value of the first touch coordinate can be increased by 2 pixels in the B direction to obtain the second touch coordinate.

[0184] In an embodiment of the present application, when the acceleration direction and the trajectory direction are consistent, the user's touch operation may not be accurate due to inertia. At this time, by increasing the touch coordinates, the displacement of the touch operation can be appropriately increased, the accuracy of the touch operation can be improved, and the user experience can be smoother.

[0185] Corresponding to the display method described in the above embodiment, Figure 9 A structural block diagram of a first display device applied to a terminal device and a second display device applied to a display device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0186] Reference Figure 9 The first display device includes: a trigger module 9110, a touch position determination module 9120 and a sending module 9130.

[0187] A triggering module 9110 is configured to trigger a control event corresponding to a touch operation input in the touch area in response to the touch operation;

[0188] A touch position determination module 9120 is configured to determine first touch position information associated with the touch operation;

[0189] The sending module 9130 is used to determine the touch type of the touch operation based on the control event, and send the first touch position information and the touch type to the display device, so that the display device generates a touch event based on the first touch position information and the touch type, and triggers the application to respond to the touch event.

[0190] The second display device includes: a receiving module 9210, a conversion module 9220 and a packaging distribution module 9230.

[0191] Receiving module 9210, configured to receive touch information sent by a terminal device; the terminal device is configured with a touch area, and the touch information includes: a touch type to which a touch operation belongs, and first touch position information associated with the touch operation, the first touch position information being used to represent a touch position within the touch area associated with the touch operation;

[0192] A conversion module 9220, configured to convert the first touch position information into second touch position information representing a position within the user interface;

[0193] The encapsulation distribution module 9230 is configured to generate a touch event according to the second touch position information and the touch type, and distribute the touch event to an application currently displayed in the user interface to trigger the application to respond to the touch event.

[0194] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0195] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a controller, the steps in the above-mentioned various method embodiments can be implemented.

[0196] An embodiment of the present application provides a computer program product. When the computer program product is run on a display device, a controller can implement the steps of the display method described in each of the above embodiments when the computer program product is executed.

[0197] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, mobile hard disk, magnetic disk or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0198] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0199] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0200] In the embodiments provided in this application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0201] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0202] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A display device, characterized in that: include: a display configured to display a user interface; The controller is configured as: Receive touch information sent by the terminal device; The terminal device is configured with a touch area, and the touch information includes: a touch type to which a touch operation belongs, and first touch position information associated with the touch operation, the first touch position information being used to represent a touch position associated with the touch operation within the touch area; converting the first touch position information into second touch position information for representing a position within the user interface; A touch event is generated according to the second touch position information and the touch type, and the touch event is distributed to an application currently displayed in the user interface to trigger the application to respond to the touch event.

2. The display device according to claim 1, wherein Before the controller converts the first touch position information into second touch position information for representing a position within the user interface, the controller is further configured to: receiving first screen information sent by the terminal device, and acquiring second screen information of the display; determining a coordinate scaling ratio according to the first screen information and the second screen information; When the controller converts the first touch position information into second touch position information representing a position within the user interface, the controller is configured to: The first touch position information is converted into the second touch position information according to the coordinate scaling ratio.

3. The display device according to claim 1 or 2, characterized in that Before the controller generates a touch event according to the second touch position information and the touch type, the controller is further configured to: Correcting the second touch position information to obtain corrected touch position information; When the controller generates a touch event according to the second touch position information and the touch type, the controller is configured to: The touch event is generated according to the corrected touch position information and the touch type.

4. The display device according to claim 3, wherein When the controller performs the correction of the second touch position information, the controller is configured to: The second touch position information is corrected by a preset signal processing method, wherein the preset signal processing method includes at least one of the following: signal smoothing processing, signal filtering processing, signal interpolation processing, and signal deletion processing.

5. The display device according to claim 3, wherein When the controller generates the touch event according to the corrected touch position information and the touch type, the controller is configured to: Create an input event; The event attribute of the input event is set according to the corrected touch position information and the touch type to obtain the touch event.

6. The display device according to claim 1 or 2, characterized in that Each application currently displayed in the user interface is provided with an independent application window; when the controller executes the step of distributing the touch event to the application currently displayed in the user interface, the controller is configured to: determining a target window according to the second touch position information, wherein the target window is an application window displayed on the top layer of the user interface and including the second touch position information; Distribute the touch event to the application corresponding to the target window.

7. A terminal device, characterized in that: include: a display configured to display a touch area; The controller is configured as: In response to a touch operation input in the touch area, triggering a control event corresponding to the touch operation; Determining first touch position information associated with the touch operation; Determine the touch type of the touch operation according to the control event, send the first touch position information and the touch type to the display device, so that the display device generates a touch event according to the first touch position information and the touch type, and triggers an application to respond to the touch event.

8. The terminal device according to claim 7, characterized in that Before the controller determines the first touch position information associated with the touch operation, the controller is further configured to: Acquiring movement data of the terminal device, wherein the movement data includes acceleration direction and acceleration magnitude; Determining trajectory data of a touch trajectory corresponding to the touch operation, wherein the trajectory data includes a trajectory direction; When the acceleration magnitude is greater than or equal to a preset acceleration threshold, the controller is configured to: The first touch position information associated with the touch operation is determined according to the acceleration direction and the trajectory direction.

9. The terminal device according to claim 8, characterized in that When the controller determines the first touch position information associated with the touch operation according to the acceleration direction and the trajectory direction, the controller is configured to: In a case where the acceleration direction and the trajectory direction are inconsistent, the first touch coordinates when the touch operation is triggered are determined as the first touch position information associated with the touch operation.

10. The terminal device according to claim 9, characterized in that When the controller determines the first touch position information associated with the touch operation according to the acceleration direction and the trajectory direction, the controller is configured to: When the acceleration direction is consistent with the trajectory direction, adjusting the first touch coordinate currently triggered by the touch operation according to the magnitude of the acceleration to obtain a second touch coordinate; The second touch coordinates are determined as first touch position information associated with the touch operation.

Images