Display device and terminal control method

Abstract

Some embodiments of the present application show a display device and a terminal control method, the method comprising: after establishing a screen projection connection with a first terminal, controlling a display to display a first screen projection interface in a user interface, the first terminal supporting a reverse control function, the first terminal comprising a second touch screen; in response to a touch instruction input by a user based on the first touch screen, obtaining a first touch track of the touch instruction; obtaining a first display range of the first screen projection interface in the user interface; determining a second touch track based on the touch points in the first display range; performing position conversion on the touch points in the second touch track to generate a third touch track, wherein the third touch track corresponds to the second touch screen; and sending the third touch track to the first terminal, so that the first terminal executes an operation corresponding to the third touch track. When the user touch area relates to the boundary area of the display device, the user instruction can still be sent to the terminal, and the user experience is improved.

Description

[0001] This application is a divisional application. The application number of the original application is 202210854818.4, and the original application date is July 18, 2022. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the technical field of display devices, and in particular to a display device and a terminal control method. Background Art

[0003] Smart Android TVs and new display terminals continue to emerge, and screen sharing between multiple screens has become a mainstream. Most screen sharing is achieved through screen projection through protocols. If the projection display interface can be touched, the projection device, such as a mobile phone, pad or other terminal, can be controlled on the projection display interface. During the control process, the boundary area of ​​the projection device may not respond, which affects the user's control and experience. Summary of the invention

[0004] Some embodiments of the present application provide a display device and a terminal control method. When the user touches an area involving a boundary area of ​​the display device, the user command can still be sent to the terminal, thereby improving the user experience.

[0005] In a first aspect, some embodiments of the present application provide a display device, including:

[0006] A display configured to display a user interface, the display comprising a first touch screen;

[0007] A communicator, configured to establish a screen projection connection with the first terminal;

[0008] The controller is configured as:

[0009] After establishing a screen projection connection with the first terminal, controlling the display to display a first screen projection interface of the first terminal in the user interface, the first terminal supports a reverse control function, the reverse control function is used to enable the user to control the first terminal by sending instructions to the display device, and the first terminal includes a second touch screen;

[0010] In response to a touch control instruction input by a user based on a first touch screen, acquiring a first touch control track of the touch control instruction, wherein the first touch control track includes at least one touch control point;

[0011] Obtain a first display range of the first projection interface in the user interface;

[0012] determining a second touch track based on the touch point within the first display range;

[0013] Converting the touch points in the second touch track to generate a third touch track, wherein the third touch track corresponds to the second touch screen;

[0014] The third touch track is sent to the first terminal, so that the first terminal performs an operation corresponding to the third touch track.

[0015] In some embodiments, the information of the touch point includes position information and a touch event, and the touch event includes a drop event, a move event, and a lift event; the controller performs the step of determining the second touch track based on the touch point within the first display range, and is further configured as follows:

[0016] According to the position information of the touch points in the first touch track, the touch points that are not in the first display range are removed to obtain a first touch track to be corrected;

[0017] If the touch event of the first touch point in the first touch track to be corrected is a moving event, changing the touch event of the first touch point to a falling event;

[0018] If the touch event of the last touch point in the first touch track to be corrected is a move event, changing the touch event of the last touch point to a lift event;

[0019] A second touch track is generated according to the first touch track to be modified after the touch event is modified.

[0020] In some embodiments, the communicator is further configured to establish a communication connection with a second terminal, and the second terminal includes a third touch screen; the controller is configured to:

[0021] After establishing a screen projection connection with the second terminal, the display is controlled to display the second screen projection interface of the second terminal in the user interface, and the first screen projection interface does not overlap with the second screen projection interface.

[0022] In some embodiments, the controller executes a touch instruction in response to a user input based on the first touch screen, and is further configured to:

[0023] Acquire a fourth touch track of the touch command, where the fourth touch track includes at least one touch point;

[0024] Acquire a first display range of the first projection interface in the user interface and a second display range of the second projection interface in the user interface;

[0025] If the fourth touch track has touch points both within the first display range and the second display range, determine a fifth touch track based on the touch points within the first display range, and determine a seventh touch track based on the touch points within the second display range;

[0026] Converting the touch points in the fifth touch track to generate a sixth touch track, wherein the sixth touch track corresponds to the second touch screen;

[0027] sending the sixth touch track to the first terminal, so that the first terminal performs an operation corresponding to the sixth touch track;

[0028] converting the position of the touch point in the seventh touch track to generate an eighth touch track, wherein the eighth touch track corresponds to the third touch screen;

[0029] The eighth touch track is sent to the second terminal, so that the second terminal performs an operation corresponding to the eighth touch track.

[0030] In some embodiments, the controller determines the seventh touch track based on the touch point within the second display range, and is further configured to:

[0031] According to the position information of the touch points in the seventh touch track, the touch points that are not in the second display range are removed to obtain a second touch track to be corrected;

[0032] If the touch event of the first touch point in the second touch track to be corrected is a moving event, changing the touch event of the first touch point to a falling event;

[0033] If the touch event of the last touch point in the second touch track to be corrected is a move event, changing the touch event of the first touch point to a lift event;

[0034] A seventh touch track is generated according to the second touch track to be modified after the touch event is modified.

[0035] In some embodiments, the controller is further configured to:

[0036] If the number of touch points in the fifth touch track exceeds a preset number; or,

[0037] The touch points of the fifth touch track include the falling touch points or the lifting touch points in the fourth touch track, the falling touch points are touch points whose touch events are falling, and the lifting touch points are touch points whose touch events are lifting; or,

[0038] If the distance between at least one touch point in the fifth touch track and the first boundary exceeds a preset value, the touch point in the fifth touch track is converted into a sixth touch track, and the first boundary is the boundary in the first projection interface adjacent to the second projection interface.

[0039] In some embodiments, the controller is further configured to:

[0040] If the number of touch points in the seventh touch track exceeds a preset number; or,

[0041] The touch point of the seventh touch track includes the drop touch point or the lift touch point in the fourth touch track; or,

[0042] The distance between at least one touch point in the seventh touch track and the second boundary exceeds a preset value, and the second boundary is a boundary in the second projection interface adjacent to the first projection interface.

[0043] In some embodiments, the controller is configured to:

[0044] If the number of touch points in the fifth touch track does not exceed the preset number; or,

[0045] The touch points of the fifth touch track do not include the drop touch points or lift touch points in the fourth touch track, or,

[0046] If the distances between the touch points in the fifth touch track and the first boundary do not exceed the preset value, the fifth touch track is removed.

[0047] In some embodiments, the controller is configured to:

[0048] If the number of touch points in the seventh touch track does not exceed the preset number; or,

[0049] The touch points of the seventh touch track do not include the drop touch points or lift touch points in the fourth touch track, or,

[0050] If the distances between the touch points in the seventh touch track and the second boundary do not exceed the preset value, the seventh touch track is removed.

[0051] In a second aspect, some embodiments of the present application provide a terminal control method, including:

[0052] After establishing a screen projection connection with the first terminal, controlling the display to display a first screen projection interface of the first terminal in the user interface, the first terminal supports a reverse control function, the reverse control function is used to enable the user to control the first terminal by sending instructions to the display device, and the first terminal includes a second touch screen;

[0053] In response to a touch control instruction input by a user based on a first touch screen, acquiring a first touch control track of the touch control instruction, wherein the first touch control track includes at least one touch control point;

[0054] Obtain a first display range of the first projection interface in the user interface;

[0055] determining a second touch track based on the touch point within the first display range;

[0056] Converting the touch points in the second touch track to generate a third touch track, wherein the third touch track corresponds to the second touch screen;

[0057] The third touch track is sent to the first terminal, so that the first terminal performs an operation corresponding to the third touch track.

[0058] Some embodiments of the present application provide a display device and terminal control method, which obtains the touch track triggered by the user on the display device (the projected screen end), generates a new touch track from the touch points within the first display range, and then sends the touch points of the new touch track to the first terminal (projected screen end) after position conversion, so as to achieve the purpose of reverse control of the first terminal. In the embodiments of the present application, when the touch track is not entirely in the first projected screen interface, the user's intention can still be sent to the first terminal, thereby achieving control over the first terminal and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 An operation scenario between a display device and a control device according to some embodiments is shown;

[0060] Figure 2 shows a hardware configuration block diagram of a control device 100 according to some embodiments;

[0061] Figure 3 shows a hardware configuration block diagram of a display device 200 according to some embodiments;

[0062] Figure 4 shows a software configuration diagram in a display device 200 according to some embodiments;

[0063] Figure 5 A flowchart showing the steps performed by a single projection screen controller according to some embodiments is shown;

[0064] Figure 6 A schematic diagram of a user interface for projecting a mobile phone screen onto a display device according to some embodiments is shown;

[0065] Figure 7 A schematic diagram showing interaction between a display device and a first terminal according to some embodiments is shown;

[0066] Figure 8 A schematic diagram of establishing a rectangular coordinate system according to some embodiments is shown;

[0067] Fig. 9 A schematic diagram showing a user touch effect display provided according to some embodiments is shown;

[0068] Fig.10 A schematic diagram showing a user touch effect display provided according to some embodiments is shown;

[0069] Fig.11 A schematic diagram showing a user touch effect display provided according to some embodiments is shown;

[0070] Fig.12 A schematic diagram showing a user touch effect display provided according to some embodiments is shown;

[0071] Fig.13 A schematic diagram showing a user touch effect display provided according to some embodiments is shown;

[0072] Fig.14 A schematic diagram showing a user touch effect display provided according to some embodiments is shown;

[0073] Fig.15 A schematic diagram of a touch point coordinate conversion of a second touch track provided according to some embodiments is shown;

[0074] Fig.16 A flowchart showing the steps executed by two controllers under the projection screen according to some embodiments is shown;

[0075] Fig.17 A schematic diagram of a user interface for projecting screens of two mobile phones onto a display device is shown according to some embodiments;

[0076] Fig.18 A schematic diagram showing a user touch effect display provided according to some embodiments is shown;

[0077] Fig.19 A schematic diagram showing a user touch effect display provided according to some embodiments is shown;

[0078] Fig. 20 A schematic diagram showing a user touch effect display provided according to some embodiments is shown;

[0079] Fig.21 A schematic diagram showing a user touch effect display provided according to some embodiments is shown;

[0080] Fig. 22 A schematic diagram of a user touch effect display provided according to some embodiments is shown. DETAILED DESCRIPTION

[0081] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0082] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0083] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0084] The terms "comprises" and any variations thereof are intended to cover but not exclude inclusions, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0085] The display device provided in the embodiments of the present application may have various implementation forms, for example, it may be a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 and Figure 2 This is a specific implementation of the display device of the present application.

[0086] Figure 1 FIG. 1 is a schematic diagram of an operation scenario between a display device and a control device according to an embodiment. Figure 1 As shown, the user can operate the display device 200 through the smart device 300 or the control apparatus 100 .

[0087] In some embodiments, the control device 100 may be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication or Bluetooth protocol communication, and other short-range communication methods, and the display device 200 is controlled wirelessly or wired. The user may control the display device 200 by inputting user commands through buttons on the remote controller, voice input, control panel input, etc.

[0088] In some embodiments, a smart device 300 (such as a mobile terminal, a tablet computer, a computer, a laptop computer, etc.) may also be used to control the display device 200. For example, the display device 200 is controlled using an application running on the smart device.

[0089] In some embodiments, the display device may not use the above-mentioned smart device or control device to receive instructions, but may receive user control through touch or gestures.

[0090] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the user's voice command control can be directly received through a module for obtaining voice commands configured inside the display device 200, or the user's voice command control can be received through a voice control device set outside the display device 200.

[0091] In some embodiments, the display device 200 also communicates data with the server 400. The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 may provide various content and interactions to the display device 200. The server 400 may be one cluster or multiple clusters, and may include one or more types of servers.

[0092] Figure 2 Schematically shows a block diagram of a configuration of the control device 100 according to an exemplary embodiment. Figure 2 As shown, the control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive input operation instructions from the user, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, playing the role of an interactive intermediary between the user and the display device 200.

[0093] like Figure 3 The display device 200 includes at least one of a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface.

[0094] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input / output.

[0095] The display 260 includes a display screen component for presenting images, and a driving component for driving image display, which is used to receive image signals output from the controller, and display video content, image content, and menu control interface components and user control UI interface.

[0096] The display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.

[0097] The display 260 also includes a touch screen, which is used to receive control instructions input by actions such as sliding or clicking a user's finger on the touch screen.

[0098] The communicator 220 is a component for communicating with an external device or server according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, and other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 200 can establish transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0099] The user interface can be used to receive control signals from the control device 100 (such as an infrared remote controller, etc.).

[0100] The detector 230 is used to collect signals from the external environment or the external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

[0101] The external device interface 240 may include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or digital high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It may also be a composite input / output interface formed by the above multiple interfaces.

[0102] The tuner-demodulator 210 receives broadcast television signals via wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals.

[0103] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0104] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 can perform operations related to the object selected by the user command.

[0105] In some embodiments, the controller includes a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM Random Access Memory (RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus (Bus), etc.

[0106] The user may input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command through a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.

[0107] "User interface" is the medium interface for interaction and information exchange between applications or operating systems and users. It realizes the conversion between the internal form of information and the form acceptable to users. The commonly used form of user interface is the Graphical User Interface (GUI), which refers to the user interface related to computer operation displayed in a graphical way. It can be an interface element such as an icon, window, control, etc. displayed on the display screen of an electronic device, where the control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc.

[0108] See also Figure 4 In some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and the kernel layer.

[0109] In some embodiments, at least one application is running in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. provided by the operating system, or applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.

[0110] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.

[0111] like Figure 4 As shown, in the embodiment of the present application, the application framework layer includes managers, content providers, etc., wherein the manager includes at least one of the following modules: an activity manager (ActivityManager) is used to interact with all activities running in the system; a location manager (Location Manager) is used to provide system services or applications with access to system location services; a package manager (Package Manager) is used to retrieve various information related to the application package currently installed on the device; a notification manager (NotificationManager) is used to control the display and clearing of notification messages; a window manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers, and desktop components on the user interface.

[0112] In some embodiments, the activity manager is used to manage the life cycle of each application and the common navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes (for example, reducing the display window, shaking the display, distorting the display, etc.).

[0113] In some embodiments, the system runtime layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system will run the C / C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

[0114] In some embodiments, the kernel layer is a layer between hardware and software. Figure 4 As shown, the kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0115] Screen sharing is mostly achieved through screen projection through protocols. If the projection display interface can be controlled by touch, the projection device, such as a mobile phone, pad or other terminal, can be controlled on the projection display interface. During the control process, the boundary area of ​​the projection device may not respond, which affects the user's control and experience.

[0116] In order to solve the above technical problems, the present application provides a display device 200. The structure and functions of each part of the display device 200 can refer to the above embodiments. In addition, based on the display device 200 shown in the above embodiments, this embodiment further improves some functions of the display device 200, such as Figure 5 As shown, the controller 250 performs the following steps:

[0117] Step S501: receiving a screen projection request sent by a first terminal, and establishing a screen projection connection with the first terminal;

[0118] The first terminal includes a display device, a mobile phone, a tablet computer, and a computer. The first terminal is connected to the display device through the same network communication protocol to perform protocol operations related to screen projection. Through the protocol layer agreement, the communication process carries relevant protocol information. The first terminal includes a second touch screen, and the first terminal supports user touch operations.

[0119] In some embodiments, when the user needs to project the interface of the first terminal to a display device, he / she will click on the screen projection on the operation interface of the first terminal, and the operation interface will display the screen projection devices connected to the same network as the first terminal. When the user selects the display device to which he / she wants to project the screen, the first terminal will send a screen projection request to the display device. After receiving the screen projection request sent by the first terminal, the screen projection service of the display device establishes a screen projection connection with the first terminal through protocol interaction.

[0120] Step S502: Control the display to display a first screen projection interface on the user interface, where the first screen projection interface is the user interface of the first terminal;

[0121] After establishing a screen projection connection with the first terminal, through protocol interaction, the user interface of the first terminal is first projected, that is, the first terminal captures the current user interface in real time and transmits it to the display device. After receiving the transmission data, the controller controls the display to display the captured user interface. The display includes a first touch screen, and the display supports user touch operation.

[0122] In some embodiments, the aspect ratio of the user interface of the first terminal is the same as that of the user interface of the display device, and the user interface of the first terminal can be completely covered on the display screen of the display in equal proportion when displayed.

[0123] In some embodiments, if the aspect ratio of the user interface of the first terminal does not match the aspect ratio of the user interface of the display device, the width and height of the user interface of the first terminal are enlarged according to a preset ratio and displayed in the center, and the portion of the first terminal user interface that is not displayed is set to a background color, such as black or white. The width ratio and height ratio of the first terminal user interface and the display device user interface can also be determined, and the smaller of the width ratio and the height ratio is selected to enlarge the width and height of the first terminal user interface and display it in the center, and the portion of the first terminal user interface that is not displayed is set to a background color, such as black or white. Exemplarily, the effect of projecting the user interface of a mobile phone onto the display of a display device is as follows: Figure 6 shown.

[0124] Step S503: confirming with the first terminal that a reverse control function is supported, so that the user can control the first terminal by sending instructions to the display device;

[0125] After the display shows the first projection interface, the first terminal interacts with the display device through a protocol to confirm whether the display device supports the reverse control function, and then inputs a command on the display device to control the first terminal. The display device sends information that it supports the reverse control function to the first terminal, and the first terminal performs corresponding initialization preparation operations after receiving the information, so that the first terminal can receive the command input by the user on the display device and perform the corresponding operation.

[0126] In some embodiments, Figure 7 As shown, the first terminal (screen projection end) is the client of the screen projection protocol and the client of the control processing, and the display device (display end) is the server of the screen projection protocol and the server of the control processing. The client of the screen projection protocol and the server of the screen projection protocol establish a connection through protocol interaction. The first terminal captures the user interface (screen recording) in real time and sends it to the display device to display the screen projection interface through the protocol transmission data stream. The client of the control processing and the server of the control processing determine the control method through control interaction processing.

[0127] Step S504: in response to a touch control instruction input by the user based on the first touch screen, obtaining a first touch control track of the touch control instruction;

[0128] In some embodiments, the user's finger or other identifiable part on the touch screen, hereinafter referred to as the finger, can touch the first touch screen to input a command. The controller can obtain a first touch track formed by the user's finger moving on the first touch screen, and the first touch track includes at least one touch point, and each touch point carries its corresponding position information and touch event. It should be noted that the user operations of this application are all single-finger operations on the first touch screen.

[0129] In some embodiments, a rectangular coordinate system is established with two adjacent sides of the first touch screen as the x-axis and y-axis and the intersection of the two adjacent sides of the first touch screen as the coordinate origin, such as Figure 8 The position information of each touch point is determined by the coordinate point where it is located. For example, the position information of the touch point can be represented by (x, y).

[0130] Touch events include down events, move events, and up events. If a touch point carries a down event, it means that the user presses the finger at the touch point. If a touch point carries a move event, it means that the finger is moved at the touch point. If a touch point carries a lift event, it means that the finger is lifted at the touch point.

[0131] The first touch track includes at least one touch point. A complete touch track consists of a drop event, a move event, and a lift event. Ideally, a touch point can carry three events at the same time, that is, the user's finger performs press, move, and lift operations on the same touch point. But in reality, the user's finger inevitably moves slightly during the manipulation process, so the touch track consists of a drop event, a series of move events, and a lift event.

[0132] Step S505: Obtain a first display range of the first projection interface in the user interface;

[0133] In some embodiments, Figure 8 The first display range of the first projection interface in the user interface can be determined in the rectangular coordinate system shown. The first display range can be represented by the coordinates of the diagonal points in the first projection interface. For example: A1 (min_x, min_y) and A2 (max_x, max_y), A1 and A2 are diagonal points to each other.

[0134] Step S506: determining a second touch track based on the touch point within the first display range;

[0135] In some embodiments, the step of determining the second touch track includes:

[0136] Acquire coordinate information of a touch point in the first touch track;

[0137] Determining whether the touch point is within the first display range according to the position information of the touch point;

[0138] Among them, whether the touch point is within the first display range can be determined by judging whether the horizontal coordinate and vertical coordinate of the touch point are between the horizontal coordinate and vertical coordinate of the two diagonal points of the first projection interface. For example: the coordinates of touch point B are (x1, y1), if min_x≤x1≤max_x and min_y≤y1≤max_y, then touch point B is within the first display range. If min_x>x1 or x1>max_x, and / or, min_y>y1 or y1>max_y, then touch point B is not within the first display range.

[0139] If the touch point is not within the first display range, remove the touch point information;

[0140] If the touch point is within the first display range, the touch point information is retained.

[0141] After each touch point in the first touch track is judged, the remaining touch point information constitutes the first touch track to be corrected.

[0142] If the touch event of the first touch point in the first touch track to be corrected is a moving event, changing the touch event of the first touch point to a falling event;

[0143] If the touch event of the first touch point in the first touch track to be corrected is a drop event, no modification is made.

[0144] If the touch event of the last touch point in the first touch track to be corrected is a move event, changing the touch event of the last touch point to a lift event;

[0145] If the touch event of the first touch point in the first touch track to be corrected is a lift event, no modification is made.

[0146] It should be noted that, as the touch points generated by the user's finger sliding on the touch screen are arranged in chronological order, the first touch point and the last touch point in the first touch track to be corrected can be directly determined in the information recording the touch track.

[0147] After the first touch track to be corrected is modified, it becomes the second touch track.

[0148] For example, Fig. 9 As shown, the areas touched by the user are all within the first display range, that is, the coordinates of the first touch trajectory [A(down)-B(move)-C(move)-D(move)-E(up)] are all within the first display range, then the second touch trajectory is determined to be [A(down)-B(move)-C(move)-D(move)-E(up)].

[0149] like Fig.10As shown, according to the original touch action of the user, the touch area is partially within the first display range and partially outside the first display range, that is, in the first touch track [A(down)-B(move)-C(move)-D(move)-E(move)-F(move)-G(up)], A / B / C is outside the first display range, and D / E / F / G is inside the first display range. The touch point information that is not within the first display range is removed, and the first touch track to be corrected is obtained as [D(move)-E(move)-F(move)-G(up)], and the second touch track is generated as [D(down)-E(move)-F(move)-G(up)]. The second touch track is the same as Fig.10 The touch mapping actions correspond to the

[0150] like Fig.11 As shown, according to the original touch action of the user, the touch area is partially within the first display range and partially outside the first display range, that is, in the first touch track [A(down)-B(move)-C(move)-D(move)-E(move)-F(move)-G(up)], D / E / F / G are outside the first display range, and A / B / C are inside the first display range. The touch point information that is not within the first display range is removed, and the first touch track to be corrected is obtained as [A(down)-B(move)-C(move)], and the second touch track is generated as [A(down)-B(move)-C(up)]. The second touch track is the same as Fig.11 The touch mapping actions correspond to the

[0151] like Fig.12 As shown, the coordinates of the upper left corner A1 of the first projection interface are (701, 5), and the coordinates of the lower right corner A2 are (1218, 1075). The information of each touch point includes (x, y, key). According to the original touch action of the user, the touch area is partially within the first display range and partially outside the first display range, that is, the first touch trajectory can be expressed as:

[0152] ((x=700.3, y=647.0833, down),

[0153] (x=720.8, y=650.80246, move),

[0154] (x=700.3, y=651.80246, move),

[0155] (x=702.3, y=652.80246, move),

[0156] (x=730.3, y=651.1, move),

[0157] (x = 750.3, y = 652.80246, move) ......

[0159] (x = 1217.3, y = 652.80246, move)

[0160] (x = 1219.3, y = 653.80246, move)

[0161] (x = 1412.0, y = 654.14832, up))

[0162] Determine the range of the first display range according to the coordinates of A1 and A2, that is, min_x is 701, max_x is 1218, min_y is 5, and max_y is 1075.

[0163] Taking the third touch point as an example (x = 700.3, y = 651.80246, move), 700.3 < min_x, so the third touch point is outside the screen projection area and is discarded. For the fourth touch point (x = 702.3, y = 652.80246, move), max_x > 702.3 > min_x, max_y > 652.80246 > min_y, so it is retained.

[0164] The following is the first trajectory to be corrected:

[0165] ((x = 702.3, y = 652.80246, move),

[0166] (x = 730.3, y = 651.1, move),

[0167] (x = 750.3, y = 652.80246, move) ......

[0169] (x = 1217.3, y = 652.80246, move))

[0170] Since each complete touch trajectory consists of a series of down - move - up, the following second touch trajectory is generated:

[0171] ((x = 702.3, y = 652.80246, down),

[0172] (x = 730.3, y = 651.1, move),

[0173] (x = 750.3, y = 652.80246, move) ......

[0175] (x=1217.3, y=652.80246, up))

[0176] Its second touch track is Fig.12 The corresponding touch mapping actions.

[0177] In some embodiments, after removing the touch points that are not within the first display range, the remaining touch point information may form more than one touch track. In the case of at least two touch tracks, both touch tracks may be removed, and no operation related to the user's touch operation may be performed; one of the touch tracks may be selected, that is, one of the touch tracks may be determined as the first touch track to be corrected, and both touch tracks may be used as the first touch tracks to be corrected, and in subsequent steps, both first touch tracks to be corrected may be processed and sent to the first terminal simultaneously or successively.

[0178] In some embodiments, taking two touch tracks as an example, a first candidate touch track and a second candidate touch track are distinguished by naming them, and a method for determining a first touch track to be corrected includes:

[0179] The number of touch points of the first candidate touch track and the second candidate touch track is counted, and if the number of touch points of the first candidate touch track is greater than the number of touch points of the second touch track, the first candidate touch track is determined to be the first touch track to be corrected. If the number of touch points of the first candidate touch track is less than or equal to the number of touch points of the second touch track, the second candidate touch track is determined to be the first touch track to be corrected.

[0180] In some embodiments, the method for determining the first touch track to be corrected includes:

[0181] It is determined whether the first candidate touch track includes a falling touch point, where the falling touch point is a touch point whose touch event is a falling event. If the first candidate touch track includes the falling touch point, the first candidate touch track is determined to be the first touch track to be corrected.

[0182] If the first candidate touch track does not include the falling touch point, determine whether the second candidate touch track includes the falling touch point; if the second candidate touch track includes the falling touch point, determine the second candidate touch track as the first touch track to be corrected.

[0183] If the second candidate touch track does not include the falling touch point, the first candidate touch track and the second candidate touch track are removed, and the operation related to the user's touch operation is not performed.

[0184] In some embodiments, the method for determining the first touch track to be corrected includes:

[0185] Determine whether the first candidate touch track includes a lift-up touch point, where the lift-up touch point is a touch point where the touch event is a lift-up event;

[0186] If the first candidate touch track includes the lifted touch point, the first candidate touch track is determined to be the first touch track to be corrected.

[0187] If the first candidate touch track does not include the lift-off touch point, determining whether the second candidate touch track includes the lift-off touch point;

[0188] If the second candidate touch track includes the lifted touch point, the second candidate touch track is determined to be the first touch track to be corrected.

[0189] If the second candidate touch track does not include the touch point being lifted, the first candidate touch track and the second candidate touch track are removed, and operations related to the user's touch operation are no longer performed.

[0190] For example, Fig.13As shown, according to the original touch action of the user, the touch area is partially within the first display range and partially outside the first display range, that is, in the first touch track [A(down)-B(move)-C(move)-D(move)-E(move)-F(move)-G(up)], D is outside the first display range, and A / B / C / E / F / G are inside the first display range. The touch point information that is not in the first display range is removed, and the first candidate touch track is [A(down)-B(move)-C(move)], and the second candidate touch track is [E(move)-F(move)-G(up)]. When two touch tracks are obtained, both touch tracks are removed according to the interaction definition 1, and the operations related to the user's touch operation are no longer performed. The touch mapping action is shown in effect 1. According to interaction definition 2, the touch track containing the falling touch point needs to be retained, that is, the first touch track to be corrected is [A(down)-B(move)-C(move)], and the second touch track is [A(down)-B(move)-C(up)], and the touch mapping action is shown in effect 2. According to interaction definition 3, the touch track containing the lifting touch point needs to be retained, that is, the first touch track to be corrected is [E(move)-F(move)-G(up)], and the second touch track is [E(down)-F(move)-G(up)], and the touch mapping action is shown in effect 3. According to interaction definition 4, both touch tracks are retained. The first touch track to be corrected is [A(down)-B(move)-C(move)] and [E(move)-F(move)-G(up)], and the second touch track is [A(down)-B(move)-C(up)] and [E(down)-F(move)-G(up)]. The touch mapping action is shown in effect 4, where the interaction definition refers to the determination method selected when two touch tracks appear.

[0191] For example, Fig.14As shown, according to the original touch action of the user, the touch area is partially within the first display range and partially outside the first display range, that is, in the first touch track [A(down)-B(move)-C(move)-D(move)-E(move)-F(move)-G(move)-H(move)-I(up)], A / E / I are outside the first display range, and B / C / D / F / G / H are inside the first display range. The touch point information that is not in the first display range is removed, and the first candidate touch track is [B(move)-C(move)-D(move)], and the second candidate touch track is [E(move)-F(move)-G(move)]. When two touch tracks are obtained, according to interaction definition 1, both touch tracks have no falling touch points or lifting touch points, the two touch tracks are removed, and no operations related to the user's touch operations are performed. The touch mapping action is as shown in effect 1. According to interaction definition 2, both touch tracks are retained, and the touch mapping action is as shown in effect 2.

[0192] Step S507: converting the positions of the touch points in the second touch track into a third touch track, wherein the third touch track corresponds to the second touch screen.

[0193] The coordinate information of the touch point in the second touch track is determined based on the rectangular coordinate system established by the display device. If the touch point in the second touch track is directly sent to the first terminal, the first terminal cannot recognize it. Therefore, coordinate conversion is required so that the first terminal can recognize the touch track. It should be noted that during the position conversion process, only the position information of the touch point in the second touch track is modified, and the touch event is not changed.

[0194] In some embodiments, the step of converting the position of the touch point in the second touch track to generate the third touch track includes:

[0195] Determine two adjacent display boundary values ​​of the first projection interface, which are an x-axis boundary value and a y-axis boundary value, respectively, where the display boundary values ​​are determined by the coordinates of the display boundary of the first projection interface in the rectangular coordinate system of the display device;

[0196] Calculate the difference between the original coordinates of the touch point and its corresponding boundary value to obtain the converted coordinates. Calculate the difference between the x value of the touch point and the x-axis boundary value to obtain the x value of the converted touch point. Calculate the difference between the y value of the touch point and the y-axis boundary value to obtain the y value of the converted touch point.

[0197] For example, Fig.15As shown, the left boundary of the first projection interface can be represented as x=701 in the rectangular coordinate system of the display device, and the upper boundary of the first projection interface can be represented as y=5 in the rectangular coordinate system of the display device. Therefore, the x-axis boundary value is determined to be 701 and the y-axis boundary value is 5. If the information of one of the touch points A in the second touch track is A(x=730.3, y=651.1, move), then the touch point can be represented as (x=29.3, y=646.1, move) in the third touch track.

[0198] Step S508: sending the third touch track to the first terminal, so that the first terminal performs an operation corresponding to the third touch track.

[0199] In some embodiments, after obtaining the third touch track, the controller sends the third touch track to the first terminal. After obtaining the third touch track, the first terminal processes the third touch track and executes an operation corresponding to the third touch track.

[0200] For example: The third touch track is:

[0201] ((x=584.3784, y=498.96857, down),

[0202] (x=589.3784, y=494.96857, move), ......

[0204] (x=680.3784, y=490.96857, move),

[0205] (x=696.28564, y=488.3564, up)

[0206] The coordinates of the dropped touch point and the lifted touch point are used to calculate the x-direction difference of 696.28564-584.3784=111.90723, and the y-direction difference of 498.96857-488.3564=10.612152. If the x-direction difference is greater than the y-direction difference, the direction of the third touch track is lateral movement. The x value of the dropped touch point and the x value of the lifted touch point are compared. If the x value of the dropped touch point is less than the x value of the lifted touch point, the third touch track is sliding to the right. The first terminal performs a right sliding operation.

[0207] In the above embodiment, after acquiring the entire first touch track, the first touch track is processed to obtain the second touch track, and the entire second touch track is sent to the first terminal. In some embodiments, the controller can process a touch point after acquiring the touch point, and then send it to the first terminal. The specific method is as follows:

[0208] Get the coordinate information of the touch point;

[0209] Determine whether the touch point is within the first display range;

[0210] If the touch point is not within the first display range, removing the touch point;

[0211] If the touch point is within the first display range, determining whether a previous touch point associated with the touch point is within the first display range;

[0212] If the previous touch point associated with the touch point is not within the first display range, determine whether the touch event of the touch point is a drop event. If it is not a drop event, change the touch event of the touch point to a drop event. If it is a drop event, do not process the touch point, that is, retain the information of the touch point.

[0213] If the touch point has no associated previous touch point, it can be directly determined whether the touch event of the touch point is a falling event.

[0214] If a previous touch point associated with the touch point is within the first display range, determining whether a subsequent touch point associated with the touch point is within the first display range;

[0215] If a subsequent touch point associated with the touch point is not within the first display range, it is determined whether the touch event of the touch point is a lift event; if it is not a lift event, the touch event of the touch point is changed to a lift event.

[0216] If the touch point has no associated next touch point, it is possible to directly determine whether the touch event of the touch point is a lift event without processing the touch point, that is, retaining the information of the touch point.

[0217] If a subsequent touch point associated with the touch point is within the first display range, the touch point is not processed, that is, the information of the touch point is retained.

[0218] It should be noted that the retained position information of the touch point must be sent to the first terminal after coordinate conversion, so that the first terminal performs a corresponding operation after receiving the information of the touch point.

[0219] The embodiment of the present application can quickly send the touch point information to the first terminal, so that the first terminal can perform the corresponding operation after obtaining part of the touch point information. For example, when the user drags a display icon, the first terminal does not need to wait for the finger to be lifted before completing the icon drag. The first terminal can execute the drag command input by the user on the display device in real time.

[0220] In some embodiments, the user can project the user interfaces of multiple terminals onto the user interface of the display device. Fig.16 As shown, the controller 250 performs the following steps:

[0221] Step S1601: receiving a screen projection request sent by the first terminal and the second terminal, and establishing a screen projection connection with the first terminal and the second terminal;

[0222] The first terminal and the second terminal include a display device, a mobile phone, a tablet computer, and a computer. The first terminal and the second terminal are respectively connected to the display device through the same network communication protocol to perform protocol operations related to screen projection. Through the protocol layer agreement, the relevant protocol information is carried during the communication process. The first terminal includes a second touch screen, and the second terminal includes a third touch screen. The first terminal and the second terminal support user touch operations.

[0223] Step S1602: Control the display to display a first screen projection interface and a second screen projection interface on the user interface, where the first screen projection interface is the user interface of the first terminal, and the second screen projection interface is the user interface of the second terminal;

[0224] After establishing a screen projection connection with the first terminal and the second terminal, through protocol interaction, the user interfaces of the first terminal and the second terminal are first projected, that is, the first terminal and the second terminal capture the current user interface in real time and transmit it to the display device. After receiving the transmitted data, the controller controls the display to display the captured user interface.

[0225] In some embodiments, the width and height of the user interface of the first terminal are enlarged according to a first preset ratio, and the width and height of the user interface of the second terminal are enlarged according to a second preset ratio. The display positions of the first projection interface and the second projection interface do not overlap, and the first preset ratio and the second preset ratio may be the same or different. The first projection interface and the second projection interface may be arranged horizontally, and the first projection interface and the second projection interface may be arranged vertically. The portion that is not displayed for the first projection interface and the second projection interface is set to a background color, such as black or white. Exemplary, the effect of projecting the user interfaces of two mobile phones onto the display of a display device, such as Fig.17 shown.

[0226] Step S1603: confirming with the first terminal and the second terminal respectively that the reverse control function is supported, so that the user can control the first terminal and the second terminal by sending instructions to the display device;

[0227] After the display displays the first screen projection interface and the second screen projection interface, the first terminal and the second terminal respectively interact with the display device through protocols to confirm whether the display device supports the reverse control function, and then input commands on the display device to control the first terminal and the second terminal. The display device will send information that supports the reverse control function to the first terminal and the second terminal, and the first terminal and the second terminal will perform corresponding initialization preparation operations after receiving the information, so that the first terminal and the second terminal can receive the commands input by the user on the display device and perform corresponding operations.

[0228] Step S1604: in response to a touch control instruction input by the user based on the first touch screen, acquiring a fourth touch control track of the touch control instruction;

[0229] In some embodiments, the user's finger can touch and slide on the first touch screen to input instructions. The controller can obtain a fourth touch track formed by the user's finger moving on the first touch screen, and the fourth touch track includes at least one touch point, and each touch point carries its corresponding position information and touch event. It should be noted that the user operations of this application are all single-finger operations on the first touch screen.

[0230] Step S1605: obtaining a first display range of the first projection interface in the user interface and a second display range of the second projection interface in the user interface;

[0231] Step S1606: Determine whether there is a touch point within the first display range and the second display range;

[0232] If there are touch points in both the first display range and the second display range, the controller executes step S1607: determining a fifth touch track based on the touch points in the first display range, and determining a sixth touch track based on the touch points in the second display range;

[0233] In some embodiments, the steps of determining the fifth touch track and the seventh touch track are similar to the step of determining the second touch track, and are not described in detail herein.

[0234] Step S1608: determining whether the fifth touch track is a valid track;

[0235] In some embodiments, whether the fifth touch track is a valid track may be determined based on an interaction definition, and the interaction definition may be based on user habits or default settings. The step of determining whether the fifth touch track is a valid track includes:

[0236] Determining whether the number of touch points in the fifth touch track exceeds a preset value;

[0237] If the number of touch points of the fifth touch track exceeds a preset value, determining the fifth touch track as a valid track;

[0238] If the number of touch points of the fifth touch track does not exceed the preset value, the fifth touch track is determined to be an invalid track.

[0239] Exemplarily, the preset value is 20, and if the number of touch points in the fifth touch track is 30, the fifth touch track is determined to be a valid track. If the number of touch points in the fifth touch track is 5, the fifth touch track is determined to be an invalid track.

[0240] In some embodiments, the step of determining whether the fifth touch track is a valid track includes:

[0241] Determine whether there is an original falling touch point in the fifth touch track, where the original falling touch point is the falling touch point in the fourth touch track, and the falling touch point in the fourth touch track may be marked to be distinguished from other falling touch points in subsequent steps;

[0242] If there is an original falling touch point in the fifth touch track, determining the fifth touch track as a valid track;

[0243] If there is no original falling touch point in the fifth touch track, the fifth touch track is determined to be an invalid track.

[0244] In some embodiments, the step of determining whether the fifth touch track is a valid track includes:

[0245] Determine whether there is an original lift-up touch point in the fifth touch track, where the original lift-up touch point is the lift-up touch point in the fourth touch track, and the lift-up touch point in the fourth touch track may be marked to be distinguished from other lift-up touch points in subsequent steps;

[0246] If there is an original lifted touch point in the fifth touch track, determining the fifth touch track as a valid track;

[0247] If there is no original lifted touch point in the fifth touch track, determining the fifth touch track as an invalid track;

[0248] For example, Fig.18As shown, according to the original touch action of the user, the touch area is partially within the first display range and the second display range, and partially outside the first display range and the second display range, that is, the fourth touch trajectory is [A(down)-B(move)-C(move)-D(move)-E(move)-F(move)-G(move)-H(move)-I(move)-J(up)], F is outside the first display range and the second display range, A / B / C / D / E is within the first display range, and G / H / I / J is within the second display range. Mark A and J as A* and J*, remove the F touch point information, the first touch track to be corrected is [A*(down)-B(move)-C(move)-D(move)-E(move)], the fifth touch track is [A*(down)-B(move)-C(move)-D(move)-E(up)], and the second touch track to be corrected is [G(move)-H(move)-I(move)-J*(up)]. The seventh touch track is [G(down)-H(move)-I(move)-J*(up)]. According to interaction definition 1, the touch track with the original falling touch point is a valid track, and the touch mapping action is shown in effect 1. According to interaction definition 2, the touch track with the original lifting touch point is a valid track, and the touch mapping action is shown in effect 2. According to interaction definition 3, the touch track with the original falling touch point or the original lifting touch point is a valid track, and the touch mapping action is shown in effect 3.

[0249] For example, Fig.19 As shown, the user touches the original action to display the touch track through the first display range and the second display range. According to interaction definition 1, both touch tracks are valid tracks, and the touch mapping action is shown in effect 1. According to interaction definition 2, the touch track with the original falling touch point or the original lifting touch point is a valid track, and both touch tracks are invalid tracks, and the touch mapping action is shown in effect 2.

[0250] In some embodiments, in the process of determining the fifth touch track based on the touch points within the first display range, it may be determined whether there is a falling touch point in the first touch track to be corrected;

[0251] If there is a falling touch point in the first touch track to be corrected, retain the first touch track to be corrected, and correct it to obtain a fifth touch track;

[0252] If there is no falling touch point in the first touch track to be corrected, the first touch track to be corrected is removed.

[0253] In some embodiments, in the process of determining the fifth touch track based on the touch points within the first display range, it may be determined whether there is a lifted touch point in the first touch track to be corrected;

[0254] If there is a lifted touch point in the first touch track to be corrected, retain the first touch track to be corrected, and correct it to obtain a fifth touch track;

[0255] If the touch point is not lifted in the first touch track to be corrected, the first touch track to be corrected is removed.

[0256] In some embodiments, the step of determining whether the fifth touch track is a valid track includes:

[0257] It is determined that the distance between the touch point in the fifth touch track and the first boundary exceeds a preset value, and the first boundary is a boundary in the first projection interface adjacent to the second projection interface.

[0258] If the distance between at least one touch point in the fifth touch track and the first boundary exceeds a preset value, determining the fifth touch track as a valid track;

[0259] If the distances between all touch points in the fifth touch track and the first boundary do not exceed a preset value, the fifth touch track is determined to be an invalid track.

[0260] For example, Fig. 20 As shown, the diagonal points A1 (295, 5) and A2 (812, 1075) of the first projection interface, and the diagonal points C1 (1107, 5) and C2 (1624, 1075) of the first projection interface are preset to be 10% of the height or width position of the first display range. The width of the first projection interface and the height of the second projection interface are 517 and 1070, respectively.

[0261] The coordinates corresponding to the fourth touch track [A(down)-BCDEF(up)] are as follows:

[0262] A(x=833.3, y=247.0833, down),

[0263] B(x=780.8, y=260.80246, move),

[0264] C(x=829.3, y=290.80246, move),

[0265] D(x=1130.3, y=29280246, move),

[0266] E(x=1250.3, y=321.1, move),

[0267] F(x=1301.3, y=335.80246, up).

[0268] After calculation, it is found that point A / C is not within the first display range and the second display range, point B is within the first display range, and point D / E / F is within the second display range. The distance between point B and the first boundary (x=812) is d=812-780.8=31.2, 31.2÷517=6%<10%. Point B is an invalid track. The distance between point F and the first boundary (x=1107) is d=1301.3-1107=194.3, 194.3÷517=38%>10%. DEF is a valid track.

[0269] In some embodiments, the step of determining whether the fifth touch track is a valid track includes:

[0270] It is determined that the distance between the touch point in the fifth touch track and the first boundary exceeds a preset value, and the first boundary is a boundary in the first projection interface adjacent to the second projection interface.

[0271] If the distance between at least one touch point in the fifth touch track and the first boundary exceeds a preset value, the fifth touch track is a valid track;

[0272] If the distances between all touch points in the fifth touch track and the first boundary do not exceed the preset value, calculating the horizontal coordinate difference and the vertical coordinate difference between the falling touch point and the lifting touch point in the fifth touch track;

[0273] If at least one of the horizontal coordinate difference and the vertical coordinate difference is greater than the preset difference, the fifth touch track is a valid track;

[0274] If the horizontal coordinate difference and the vertical coordinate difference are both smaller than or equal to the preset difference, the fifth touch track is an invalid track.

[0275] For example, Fig.21 As shown, the fourth touch track is [A(down)-B(move)-C(move)-D(move)-E(move)-F(move)-G(up)], D is outside the first display range, A / B / C are within the first display range, and E / F / G are within the second display range. Remove the D touch point information, the fifth touch track is [A(down)-B(move)-C(up)], and the seventh touch track is [E(down)-F(move)-G(up)]. The distances of A / B / C from the first boundary (x=a) are all less than the preset value, but the difference between the ordinates of A and C is greater than the preset difference. Therefore, the fifth touch track is determined to be a valid track.

[0276] If the fifth touch trajectory is a valid trajectory, execute step S1609: generate a sixth touch trajectory by position conversion of the touch points in the fifth touch trajectory, so that the sixth touch trajectory corresponds to the second touch screen, and send the sixth touch trajectory to the first terminal, so that the first terminal performs an operation corresponding to the sixth touch trajectory.

[0277] If the fifth touch track is an invalid track, executing step S1610: removing the fifth touch track;

[0278] Step S1611: determining whether the seventh touch track is a valid track;

[0279] The step of determining whether the seventh touch track is a valid track is similar to the step of determining whether the fifth touch track is a valid track, and will not be described in detail herein.

[0280] In some embodiments, when there are two touch tracks, only one of the touch tracks is allowed to be determined as a valid track. This can be determined by comparing the number of touch points in the fifth touch track and the seventh touch track. If the number of touch points in the fifth touch track is greater than the number of touch points in the seventh touch track, the fifth touch track is determined to be a valid track and the seventh touch track is an invalid track. If the number of touch points in the fifth touch track is less than or equal to the number of touch points in the seventh touch track, the fifth touch track is determined to be an invalid track and the seventh touch track is a valid track.

[0281] In some embodiments, when there are two touch tracks, only one of the touch tracks is allowed to be determined as a valid track. This can be determined by comparing the generation time of the touch point in the fifth touch track and the seventh touch track. For example, Fig. 22 As shown, if the fifth touch track is generated earlier than the seventh touch track, the fifth touch track is determined to be a valid track, and the touch mapping action is shown in effect 1. The seventh touch track is an invalid track; if the seventh touch track is generated later than the fifth touch track, the seventh touch track is determined to be a valid track, and the touch mapping action is shown in effect 2.

[0282] If the seventh touch trajectory is a valid trajectory, execute step S1612: generate an eighth touch trajectory by position conversion of the touch points in the seventh touch trajectory, so that the eighth touch trajectory corresponds to the second touch screen, and send the eighth touch trajectory to the first terminal, so that the first terminal performs an operation corresponding to the eighth touch trajectory.

[0283] If the seventh touch track is an invalid track, executing step S1613: removing the seventh touch track.

[0284] If there is a touch point within the first display range and there is no touch point within the second display range, a fifth touch track is generated based on the touch points within the first display range, and the touch points in the fifth touch track are converted into a sixth touch track by position conversion, so that the sixth touch track corresponds to the second touch screen; and the sixth touch track is sent to the first terminal, so that the first terminal performs an operation corresponding to the sixth touch track;

[0285] If there is no touch point within the first display range and there is a touch point within the second display range, a seventh touch trajectory is generated based on the touch points within the second display range; the touch points in the seventh touch trajectory are converted into an eighth touch trajectory through position conversion, so that the eighth touch trajectory corresponds to the third touch screen; and the eighth touch trajectory is sent to the second terminal, so that the second terminal executes the operation corresponding to the eighth touch trajectory.

[0286] Some embodiments of the present application provide a terminal control method, which is applicable to a display device, the display device including a display, a communicator and a controller, the controller being configured to: after establishing a projection connection with the first terminal, control the display to display a first projection interface of the first terminal in a user interface, the first terminal supporting a reverse control function, the reverse control function being used to enable a user to control the first terminal by sending instructions to the display device, the first terminal including a second touch screen; in response to a touch instruction input by a user based on the first touch screen, obtain a first touch track of the touch instruction, the first touch track including at least one touch point; obtain a first display range of the first projection interface in the user interface; determine a second touch track based on the touch points within the first display range; convert the touch points in the second touch track to generate a third touch track, the third touch track corresponding to the second touch screen; send the third touch track to the first terminal, so that the first terminal performs the operation corresponding to the third touch track. When the user touch area involves the boundary area of ​​the display device, the user instruction can still be sent to the terminal to improve the user experience.

[0287] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0288] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A display device, characterized in that: include: a display configured to display a user interface; A communicator, configured to establish a screen projection connection with at least one terminal; A controller is coupled to the display and the communicator and is configured to: In response to receiving a screen projection request sent by a first terminal, establishing a screen projection connection with the first terminal, and controlling the display to display a first screen projection interface sent by the first terminal; wherein the first terminal has a touch function; In response to a first touch instruction input by a user, a first touch track corresponding to the first touch instruction and a first display range corresponding to the first projection interface on the display are acquired; the first touch track includes at least two touch points; determining a second touch track based on a touch point in the first touch track that is within the first display range; Transforming the position of the touch point in the second touch track to generate a third touch track; The third touch track is sent to the first terminal, so that the first terminal performs a corresponding operation based on the third touch track.

2. The display device according to claim 1, characterized in that Before the controller executes the first touch control instruction in response to the user input, the controller is further configured to: Sending a confirmation request to the first terminal; the confirmation request is used to request confirmation whether the first terminal supports a reverse control function, and the reverse control function is used to support a user to control the first terminal by inputting a touch track to a display device; Receive confirmation information sent by the first terminal; the confirmation information is used to indicate that the first terminal supports the reverse control function.

3. The display device according to claim 1, characterized in that The information of the touch point includes position information; the controller determines a second touch track based on the touch points in the first display range in the first touch track, and is further configured as follows: According to the position information of the touch points in the first touch track, the touch points that are not in the first display range are removed to obtain a first touch track to be corrected; wherein the touch points in the first touch track to be corrected are all located in the first display range; Based on the first touch trajectory to be corrected, the second touch trajectory is determined.

4. The display device according to claim 3, characterized in that The information of the touch point also includes a touch event, and the touch event includes a drop event, a move event, and a lift event; the controller determines the second touch track based on the first touch track to be corrected, and is further configured as follows: If the touch event of the first touch point in the first touch track to be corrected is a moving event, changing the touch event of the first touch point to a falling event; If the touch event of the last touch point in the first touch track to be corrected is a move event, changing the touch event of the last touch point to a lift event; The second touch track is generated according to the first touch track to be modified after the touch event is modified.

5. The display device according to any one of claims 1 to 4, characterized in that: The communicator is further configured to establish a communication connection with a second terminal, and the second terminal has a touch function; and the controller is further configured to: In response to receiving a screen projection request sent by the second terminal, a screen projection connection is established with the second terminal, and the display is controlled to display a second screen projection interface sent by the second terminal; wherein the first screen projection interface and the second screen projection interface do not overlap.

6. The display device according to claim 5, characterized in that: The controller is also configured to: In response to a second touch instruction input by a user, a fourth touch track corresponding to the second touch instruction and a second display range corresponding to the second projection interface on the display are acquired; the fourth touch track includes at least two touch points; If the fourth touch track has touch points both within the first display range and the second display range, determining a fifth touch track based on the touch points within the first display range in the fourth touch track, and determining a seventh touch track based on the touch points within the second display range in the fourth touch track; Performing position conversion on the touch point in the fifth touch track to generate a sixth touch track, and sending the sixth touch track to the first terminal, so that the first terminal performs a corresponding operation based on the sixth touch track; The touch points in the seventh touch track are converted to generate an eighth touch track, and the eighth touch track is sent to the second terminal, so that the second terminal performs a corresponding operation based on the eighth touch track.

7. The display device according to claim 6, characterized in that The controller determines a seventh touch track based on the touch points in the fourth touch track that are within the second display range, and is further configured to: According to the position information of the touch points in the fourth touch track, the touch points that are not in the second display range are removed to obtain a second touch track to be corrected; wherein the touch points in the second touch track to be corrected are all located in the second display range; The seventh touch track is determined based on the second touch track to be corrected.

8. The display device according to claim 7, characterized in that The controller determines the seventh touch track based on the second touch track to be corrected, including: If the touch event of the first touch point in the second touch track to be corrected is a moving event, changing the touch event of the first touch point to a falling event; If the touch event of the last touch point in the second touch track to be corrected is a move event, changing the touch event of the first touch point to a lift event; The seventh touch track is generated according to the second touch track to be modified after the touch event is modified.

9. The display device according to claim 6, characterized in that The controller executes the position conversion of the touch point in the fifth touch track to generate a sixth touch track, and is further configured to: If the number of touch points in the fifth touch track exceeds a preset number, position conversion is performed on the touch points in the fifth touch track to generate the sixth touch track; or, If the touch points of the fifth touch track include the falling touch points or the lifting touch points in the fourth touch track, the touch points in the fifth touch track are positionally transformed to generate the sixth touch track; wherein the falling touch points are touch points whose touch events are falling, and the lifting touch points are touch points whose touch events are lifting; or, If the distance between at least one touch point in the fifth touch track and the first boundary exceeds a preset value, the touch point in the fifth touch track is transformed to generate the sixth touch track; wherein the first boundary is a boundary in the first projection interface adjacent to the second projection interface.

10. The display device according to claim 9, characterized in that The controller is further configured to: If the number of touch points of the fifth touch track does not exceed the preset number, removing the fifth touch track; or, If the touch points of the fifth touch track do not include the drop touch points or lift touch points in the fourth touch track, then remove the fifth touch track; or, If the distances between the touch points in the fifth touch track and the first boundary do not exceed the preset value, the fifth touch track is removed.

11. The display device according to claim 6, characterized in that The controller performs the position conversion of the touch point in the seventh touch track to generate the eighth touch track, and is further configured as follows: If the number of touch points in the seventh touch track exceeds a preset number, position conversion is performed on the touch points in the seventh touch track to generate the eighth touch track; or, If the touch points of the seventh touch track include the drop touch points or the lift touch points in the fourth touch track, position conversion is performed on the touch points in the seventh touch track to generate the eighth touch track; or, If the distance between at least one touch point in the seventh touch track and the second boundary exceeds a preset value, the touch point in the seventh touch track is transformed to generate the eighth touch track; wherein the second boundary is a boundary in the second projection interface adjacent to the first projection interface.

12. The display device according to claim 11, characterized in that The controller is further configured to: If the number of touch points of the seventh touch track does not exceed the preset number, removing the seventh touch track; or, If the touch points of the seventh touch track do not include the drop touch points or lift touch points in the fourth touch track, then remove the seventh touch track; or, If the distances between the touch points in the seventh touch track and the second boundary do not exceed the preset value, the seventh touch track is removed.

13. A display device, characterized in that: include: a display configured to display a user interface; A communicator, configured to establish a screen projection connection with at least one terminal; A controller is coupled to the display and the communicator and is configured to: In response to receiving a screen projection request sent by a first terminal and a second terminal, respectively establishing a screen projection connection with the first terminal and the second terminal, and controlling the display to display a first screen projection interface sent by the first terminal and a second screen projection interface sent by the second terminal; wherein the first terminal and the second terminal have a touch function; In response to a touch command input by a user, a fourth touch track corresponding to the touch command, a first display range corresponding to the first projection interface on the display, and a second display range corresponding to the second projection interface on the display are acquired; the fourth touch track includes at least two touch points; If the fourth touch track has touch points both within the first display range and the second display range, determining a fifth touch track based on the touch points within the first display range in the fourth touch track, and determining a seventh touch track based on the touch points within the second display range in the fourth touch track; Performing position conversion on the touch point in the fifth touch track to generate a sixth touch track, and sending the sixth touch track to the first terminal, so that the first terminal performs a corresponding operation based on the sixth touch track; The touch points in the seventh touch track are converted to generate an eighth touch track, and the eighth touch track is sent to the second terminal, so that the second terminal performs a corresponding operation based on the eighth touch track.

14. The display device according to claim 13, characterized in that Before the controller executes the touch control instruction in response to the user input, the controller is further configured to: Sending a confirmation request to the first terminal; the confirmation request is used to request confirmation whether the first terminal supports a reverse control function, and the reverse control function is used to support a user to control the first terminal by inputting a touch track to a display device; Receive confirmation information sent by the first terminal; the confirmation information is used to indicate that the first terminal supports the reverse control function.

15. The display device according to claim 13, characterized in that The controller determines a seventh touch track based on the touch points in the fourth touch track that are within the second display range, and is further configured to: According to the position information of the touch points in the fourth touch track, the touch points that are not in the second display range are removed to obtain a second touch track to be corrected; wherein the touch points in the second touch track to be corrected are all located in the second display range; The seventh touch track is determined based on the second touch track to be corrected.

16. The display device according to claim 15, characterized in that The controller determines the seventh touch track based on the second touch track to be corrected, including: If the touch event of the first touch point in the second touch track to be corrected is a moving event, changing the touch event of the first touch point to a falling event; If the touch event of the last touch point in the second touch track to be corrected is a move event, changing the touch event of the first touch point to a lift event; The seventh touch track is generated according to the second touch track to be modified after the touch event is modified.

17. The display device according to claim 13, characterized in that The controller executes the position conversion of the touch point in the fifth touch track to generate a sixth touch track, and is further configured to: If the number of touch points in the fifth touch track exceeds a preset number, position conversion is performed on the touch points in the fifth touch track to generate the sixth touch track; or, If the touch points of the fifth touch track include the falling touch points or the lifting touch points in the fourth touch track, the touch points in the fifth touch track are positionally transformed to generate the sixth touch track; wherein the falling touch points are touch points whose touch events are falling, and the lifting touch points are touch points whose touch events are lifting; or, If the distance between at least one touch point in the fifth touch track and the first boundary exceeds a preset value, the touch point in the fifth touch track is transformed to generate the sixth touch track; wherein the first boundary is a boundary in the first projection interface adjacent to the second projection interface.

18. The display device according to claim 17, characterized in that The controller is further configured to: If the number of touch points of the fifth touch track does not exceed the preset number, removing the fifth touch track; or, If the touch points of the fifth touch track do not include the drop touch points or lift touch points in the fourth touch track, then remove the fifth touch track; or, If the distances between the touch points in the fifth touch track and the first boundary do not exceed the preset value, the fifth touch track is removed.

19. The display device according to claim 13, characterized in that The controller performs the position conversion of the touch point in the seventh touch track to generate the eighth touch track, and is further configured as follows: If the number of touch points in the seventh touch track exceeds a preset number, position conversion is performed on the touch points in the seventh touch track to generate the eighth touch track; or, If the touch points of the seventh touch track include the drop touch points or the lift touch points in the fourth touch track, position conversion is performed on the touch points in the seventh touch track to generate the eighth touch track; or, If the distance between at least one touch point in the seventh touch track and the second boundary exceeds a preset value, the touch point in the seventh touch track is transformed to generate the eighth touch track; wherein the second boundary is a boundary in the second projection interface adjacent to the first projection interface.

20. The display device according to claim 19, characterized in that The controller is further configured to: If the number of touch points of the seventh touch track does not exceed the preset number, removing the seventh touch track; or, If the touch points of the seventh touch track do not include the drop touch points or lift touch points in the fourth touch track, then remove the seventh touch track; or, If the distances between the touch points in the seventh touch track and the second boundary do not exceed the preset value, the seventh touch track is removed.

21. A terminal control method, characterized in that: include: In response to receiving a screen projection request sent by a first terminal, establishing a screen projection connection with the first terminal, and controlling a display to display a first screen projection interface sent by the first terminal; wherein the first terminal has a touch function; In response to a first touch instruction input by a user, a first touch track corresponding to the first touch instruction and a first display range corresponding to the first projection interface on the display are acquired; the first touch track includes at least two touch points; determining a second touch track based on a touch point in the first touch track that is within the first display range; Transforming the position of the touch point in the second touch track to generate a third touch track; The third touch track is sent to the first terminal, so that the first terminal performs a corresponding operation based on the third touch track.

22. A terminal control method, characterized in that: include: In response to receiving a screen projection request sent by a first terminal and a second terminal, establishing a screen projection connection with the first terminal and the second terminal respectively, and controlling a display to display a first screen projection interface sent by the first terminal and a second screen projection interface sent by the second terminal; wherein the first terminal and the second terminal have a touch function; In response to a touch command input by a user, a fourth touch track corresponding to the touch command, a first display range corresponding to the first projection interface on the display, and a second display range corresponding to the second projection interface on the display are acquired; the fourth touch track includes at least two touch points; If the fourth touch track has touch points both within the first display range and the second display range, determining a fifth touch track based on the touch points within the first display range in the fourth touch track, and determining a seventh touch track based on the touch points within the second display range in the fourth touch track; Performing position conversion on the touch point in the fifth touch track to generate a sixth touch track, and sending the sixth touch track to the first terminal, so that the first terminal performs a corresponding operation based on the sixth touch track; The touch points in the seventh touch track are converted to generate an eighth touch track, and the eighth touch track is sent to the second terminal, so that the second terminal performs a corresponding operation based on the eighth touch track.

Images