Cross-device object dragging method and device

Abstract

The invention provides a cross-device object dragging method and device, and relates to the field of electronic devices. The use efficiency of multi-terminal cooperative use is improved, and interaction among the terminals during multi-terminal cooperative use is increased. According to the specific scheme, a second terminal displays a first cursor on an object displayed by the second terminal; the second terminal receives a dragging operation input by a user by using the input device of the first terminal, wherein the dragging operation is used for initiating dragging aiming at the object; in response to the dragging operation, the second terminal displays an animation of the object moving along with the first cursor on a display screen of the second terminal; and when determining that the object is dragged out of the edge of the display screen of the second terminal, the second terminal sends dragging data to the first terminal for continuous dragging of the first terminal.

Description

[0001] This application is a divisional application, the original application number is 202080100104.3, the original application date is December 31, 2020, and the entire contents of the original application are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of electronic devices, and in particular to a cross-device object dragging method and device. BACKGROUND

[0003] With the development of electronic technology and mobile Internet, a user can have more terminals such as mobile phones, tablet computers, personal computers (PCs), smart home devices (such as televisions), etc. Generally, the use of each terminal is relatively independent. In scenarios such as collaborative office work that require multiple terminals to be used collaboratively, the user can connect multiple terminals together for use. For example, a user has a PC and a mobile phone, and the user can connect the PC and the mobile phone together for use in a wireless or wired manner, thereby realizing collaborative office work of the PC and the mobile phone. SUMMARY

[0004] Embodiments of the present application provide a cross-device object dragging method and device. In a scenario in which multiple terminals (such as a PC and a mobile phone) are used collaboratively, the method and device realize the dragging of objects such as application icons, application interfaces, files, texts, or folders from the mobile phone to the PC by using a keyboard and mouse sharing technology, thereby improving the use efficiency of the collaborative use of the PC and the mobile phone and increasing the interaction between terminals in the collaborative use of multiple terminals.

[0005] To achieve the above object, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, the present application provides a cross-device object dragging method. The method can be applied to a second terminal, and the second terminal is connected to a first terminal. The method can include: displaying a first cursor on an object displayed on the second terminal; receiving, by the second terminal, a dragging operation input by a user using an input device of the first terminal, the dragging operation being used to initiate a drag on the object; in response to the dragging operation, displaying, by the second terminal, an animation of the object moving with the first cursor on a display screen of the second terminal; and sending, by the second terminal, dragging data to the first terminal when it is determined that the object is dragged out of an edge of the display screen of the second terminal.

[0007] As an example, the input device of the first terminal can be a mouse, a touchpad, etc. The first cursor can be a cursor displayed on the display screen of the second terminal.

[0008] According to the technical scheme, in a scenario where multiple terminals are used cooperatively, a user can use an input device such as a mouse to move an object on one terminal along with a cursor among multiple terminals used cooperatively in a dragging manner. For example, the user can use an input device of one terminal to complete dragging of an object on another terminal from the another terminal to the one terminal, thereby improving the use efficiency of the multiple-terminal cooperative use, increasing the interaction between the terminals during the multiple-terminal cooperative use, and improving the user experience.

[0009] In a possible implementation, the dragging data can be used by the first terminal to display an animation of movement of the object along with a second cursor on the display screen of the first terminal. The second cursor can be a cursor displayed on the display screen of the first terminal. By sending the dragging data to the other terminal, the other terminal can continue to display the animation of movement of the object along with the cursor on the other terminal according to the operation input by the user, thereby realizing continuation of the dragging.

[0010] In another possible implementation, the object can be an icon of an application, or the object can be a window including an interface of the application. When the object is the icon of the application, the dragging data can include the icon of the application. When the object is the window, the dragging data can include the interface of the application.

[0011] In another possible implementation, after the dragging data is sent to the first terminal, the method can further include: receiving, by the second terminal, an indication message from the first terminal, the indication message being sent by the first terminal after the user inputs a dragging release operation by using the input device of the first terminal, and being used to indicate that the dragging of the object is ended; and sending, by the second terminal, data to the first terminal, the data being used by the first terminal to display the interface of the application on the display screen of the first terminal. After the dragging of the object is ended, the corresponding data is sent to the first terminal, so that the first terminal can display the interface of the application corresponding to the dragged object on the display screen of the first terminal.

[0012] In another possible implementation, when the object is the icon of the application, after the second terminal receives the dragging operation input by the user by using the input device of the first terminal, the method can further include: creating, by the second terminal, a dragging object according to the icon of the application, the dragging object including an icon same as the icon of the application; and displaying, by the second terminal, an animation of movement of the object along with a first cursor on the display screen of the second terminal in response to the dragging operation, including: displaying, by the second terminal, an animation of movement of the dragging object along with the first cursor in response to the dragging operation. The created icon same as the dragged icon of the application can be used for dragging.

[0013] In another possible implementation, when the object is a window, before the second terminal displays the first cursor on the object displayed on the second terminal, the method can further include: displaying, by the second terminal, an interface of an application; receiving, by the second terminal, a long-press operation input by a user using an input device of the first terminal, for triggering windowing of the interface of the application, when the first cursor is displayed on a status bar of the second terminal; and displaying, by the second terminal, a window in response to the long-press operation, the window including a screenshot of the interface of the application, and the drag data including the screenshot of the interface of the application.

[0014] In another possible implementation, the displaying, by the second terminal, of the animation of the object moving with the first cursor on the display screen of the second terminal can include: displaying, by the second terminal, an animation of a shadow of the object moving with the first cursor on the display screen of the second terminal.

[0015] In another possible implementation, the object can be a text, a file, or a folder, and the drag data includes a bitmap of the drag event content and a shadow; when the object is the text, the drag event content includes the text; and when the object is the file or the folder, the drag event content is a file path.

[0016] In another possible implementation, after the displaying, by the second terminal, of the animation of the object moving with the first cursor on the display screen of the second terminal in response to the drag operation, the method can further include: obtaining and caching, by the second terminal, the drag data.

[0017] In another possible implementation, before the sending, by the second terminal, of the drag data to the first terminal, the method can further include: receiving, by the second terminal, a request message from the first terminal, the request message being used to request the drag data; and the sending, by the second terminal, of the drag data to the first terminal can include: sending, by the second terminal, the cached drag data to the first terminal.

[0018] In another possible implementation, before the displaying, by the second terminal, of the first cursor on the object displayed on the second terminal, the method can further include: receiving, by the second terminal, first shuttle state information from the first terminal, the first shuttle state information being used to indicate a start of the shuttle; and displaying, by the second terminal, the first cursor on the second terminal in response to the first shuttle state information. After receiving the shuttle state information indicating the start of the shuttle, the user can control the second terminal using the input device of the first terminal.

[0019] In another possible implementation, after the displaying, by the second terminal, of the first cursor on the second terminal, the method can further include: receiving, by the second terminal, an operation of moving the first cursor input by a user using an input device of the first terminal; and displaying, by the second terminal, an animation of the first cursor moving on the display screen of the second terminal in response to the operation of moving the first cursor, until the first cursor moves to the object. After the start of the cursor shuttle, the user can control the second terminal using the input device of the first terminal.

[0020] In another possible implementation, the drag operation can include a press operation and a move operation; the second terminal receiving the drag operation input by the user using the input device of the first terminal can include: the second terminal receiving a first operation parameter from the first terminal, the first operation parameter being an operation parameter contained in a press event received by the first terminal after the user performs the press operation using the input device of the first terminal; the second terminal receiving a second operation parameter from the first terminal, the second operation parameter being an operation parameter contained in a move event received by the first terminal after the user performs the move operation using the input device of the first terminal; the second terminal simulating the press event according to the first operation parameter and simulating the move event according to the second operation parameter, the press event and the move event being used to indicate that the drag operation for the object is initiated; and the second terminal displaying an animation of the object moving with the first cursor on the display screen of the second terminal in response to the drag operation, including: the second terminal displaying the animation of the object moving with the first cursor in response to the press event and the move event.

[0021] In another possible implementation, the second terminal receiving the operation of moving the first cursor input by the user using the input device of the first terminal can include: the second terminal receiving a third operation parameter from the first terminal, the third operation parameter being an operation parameter contained in a move event received by the first terminal after the user performs the move operation using the input device of the first terminal; and the second terminal simulating the move event according to the third operation parameter, the move event being used to indicate that the first cursor is moved.

[0022] In another possible implementation, after the second terminal determines that the object is dragged out of the edge of the display screen of the second terminal, the method can further include: the second terminal sending second shuttle state information to the first terminal, the second shuttle state information being used to indicate that the shuttle ends. By sending the shuttle state information indicating that the shuttle ends to the first terminal, the user can use the input device of the first terminal to control the first terminal.

[0023] In another possible implementation, the long-press operation can include a press operation and a lift operation; the second terminal receiving the long-press operation input by the user using the input device of the first terminal can include: the second terminal receiving a fourth operation parameter from the first terminal, the fourth operation parameter being an operation parameter contained in a press event received by the first terminal after the user performs the press operation using the input device of the first terminal when the first cursor is displayed on the status bar; the second terminal receiving a fifth operation parameter from the first terminal, the fifth operation parameter being an operation parameter contained in a lift event received by the first terminal after the user performs the lift operation using the input device of the first terminal after performing the press operation for a period of time; the second terminal simulating the press event according to the fourth operation parameter and simulating the lift event according to the fifth operation parameter, the press event and the lift event being used to trigger the interface windowing of the application; and the second terminal displaying the window in response to the long-press operation, including: the second terminal displaying the window in response to the press event and the lift event.

[0024] In another possible implementation, the method can further include: the second terminal creating a virtual input device after the connection with the first terminal is successfully established; or the second terminal receiving a notification message from the first terminal, the notification message being used to indicate that the key-mouse sharing mode of the first terminal has been started, and the second terminal creating the virtual input device in response to the notification message; wherein the virtual input device is used by the second terminal to simulate the input event according to the operation parameter. By means of the key-mouse sharing technology, the object dragging between multiple terminals using the input device of one terminal is realized.

[0025] In a second aspect, the present application provides a cross-device object dragging method, which can be applied to a first terminal connected with a second terminal, and can include: the first terminal sending, to the second terminal, data of a drag operation input by a user using an input device of the first terminal, the drag operation being used to initiate the dragging of an object displayed by the second terminal; and the first terminal receiving, from the second terminal, drag data, the drag data being sent by the second terminal to the first terminal after the second terminal determines that the object is dragged out of an edge of a display screen of the second terminal during the movement of the object along with a first cursor on the display screen of the second terminal.

[0026] As an example, the input device of the first terminal can be a mouse, a touchpad, or the like. The first cursor can be a cursor displayed on the display screen of the second terminal.

[0027] According to the technical scheme, in a scenario of cooperative use of multiple terminals, a user can use an input device such as a mouse to move an object on one terminal to another terminal in a drag-and-drop manner, thereby improving the use efficiency of the cooperative use of multiple terminals, increasing the interaction between the terminals during the cooperative use of multiple terminals, and improving the user experience.

[0028] In a possible implementation, after the first terminal receives the drag data from the second terminal, the method further includes: displaying, by the first terminal, an animation of movement of the object along the second cursor on the display screen of the first terminal according to the drag data and the drag operation. The second cursor can be a cursor displayed on the display screen of the first terminal. According to the received drag data and the operation input by the user, the animation of movement of the object along the cursor can be continuously displayed on the first terminal, thereby realizing the continuation of the drag operation.

[0029] In another possible implementation, the object can be an icon of an application, or the object can be a window including an interface of the application. When the object is the icon of the application, the drag data can include the icon of the application. When the object is the window, the drag data can include the interface of the application.

[0030] In another possible implementation, after the first terminal receives the drag data from the second terminal, the method further includes: receiving, by the first terminal, a drag release operation input by the user using the input device of the first terminal; and displaying, by the first terminal, the interface of the application on the display screen of the first terminal in response to the drag release operation. After the drag operation on the object ends, the first terminal can display the interface of the application corresponding to the dragged object on the display screen of the first terminal.

[0031] In another possible implementation, the displaying, by the first terminal, of the interface of the application on the display screen of the first terminal can include: sending, by the first terminal, an indication message to the second terminal, the indication message being used to indicate that the drag operation on the object ends; receiving, by the first terminal, data from the second terminal; and displaying, by the first terminal, the interface of the application on the display screen of the first terminal according to the data.

[0032] In another possible implementation, when the object is the window, before the first terminal sends, to the second terminal, the data of the drag operation input by the user using the input device of the first terminal, the method further includes: sending, by the first terminal, data of a long press operation input by the user using the input device of the first terminal to the second terminal, the long press operation being input by the user when the first cursor is displayed on the status bar of the second terminal and being used to trigger windowing of the interface of the application displayed by the second terminal. The window specifically includes a screenshot of the interface of the application, and the drag data specifically includes the screenshot of the interface of the application.

[0033] In another possible implementation, the first terminal displays an animation of the object moving with the second cursor on the display screen of the first terminal according to the drag data and the drag operation, which can include that the first terminal displays an animation of a shadow of the object moving with the second cursor on the display screen of the first terminal according to the drag data and the drag operation.

[0034] In another possible implementation, the object can be a text, a file or a folder; the drag data includes a drag event content and a bitmap of the shadow; when the object is the text, the drag event content includes the text; and when the object is the file or the folder, the drag event content is a file path.

[0035] In another possible implementation, before the first terminal sends the data of the drag operation input by the user using the input device of the first terminal to the second terminal, the method can further include that the first terminal determines that the second cursor slides out of the edge of the display screen of the first terminal. The user triggers the cursor switching by sliding the cursor on the display screen of the first terminal out of the edge of the display screen of the first terminal.

[0036] In another possible implementation, before the first terminal determines that the second cursor slides out of the edge of the display screen of the first terminal, the method can further include that the first terminal receives an operation of moving the second cursor input by the user using the input device of the first terminal; and in response to the operation of moving the second cursor, the first terminal displays an animation of the second cursor moving on the display screen of the first terminal.

[0037] In another possible implementation, after the first terminal determines that the second cursor slides out of the edge of the display screen of the first terminal, the method can further include that the first terminal sends first switching state information to the second terminal, the first switching state information being used to indicate the start of the switching. After the cursor switching, the switching state information used to indicate the start of the switching is sent to other terminals, so that the other terminals are ready for the subsequent drag, such as displaying the cursor.

[0038] In another possible implementation, after the first terminal determines that the second cursor slides out of the edge of the display screen of the first terminal, the method can further include that the first terminal sends data of the operation of moving the first cursor input by the user using the input device of the first terminal to the second terminal, so as to display an animation of the first cursor moving on the display screen of the second terminal. After the cursor switching, the first terminal sends the operation input by the user using the input device of the first terminal to other terminals, so as to realize the control of the input device of the first terminal on the other terminals.

[0039] In another possible implementation, the drag operation can include a pressing operation and a moving operation; the first terminal sends, to the second terminal, data of the drag operation input by the user using the input device of the first terminal, including: after the user performs the pressing operation using the input device of the first terminal, the first terminal intercepts a pressing event; the first terminal sends, to the second terminal, first operation parameters included in the pressing event; after the user performs the moving operation using the input device of the first terminal, the first terminal intercepts a moving event; the first terminal sends, to the second terminal, second operation parameters included in the moving event; wherein the first operation parameters and the second operation parameters are respectively used by the second terminal to simulate the pressing event and the moving event, and are further used to instruct to initiate the drag on the object.

[0040] In another possible implementation, the first terminal sends, to the second terminal, data of the operation of moving the first cursor input by the user using the input device of the first terminal, including: after the user performs the moving operation using the input device of the first terminal, the first terminal intercepts a moving event; the first terminal sends, to the second terminal, third operation parameters included in the moving event, the third operation parameters being used by the second terminal to simulate the moving event, and being further used to instruct to move the first cursor.

[0041] In another possible implementation, before the first terminal displays, on the display screen of the first terminal, the animation of the object moving with the second cursor according to the drag data and the drag operation, the method can further include: the first terminal receives second shuttle state information from the second terminal, the second shuttle state information being used to instruct to end the shuttle.

[0042] In another possible implementation, before the first terminal receives the drag data from the second terminal, the method can further include: the first terminal sends, to the second terminal, a request message, the request message being used to request the drag data.

[0043] In another possible implementation, after the first terminal receives the second shuttle state information from the second terminal, the method can further include: the first terminal cancels the interception of the input event; the first terminal displays, on the display screen of the first terminal, the animation of the object moving with the second cursor according to the drag operation and the drag data, including: after the user performs the moving operation using the input device of the first terminal, the first terminal receives a moving event; the first terminal generates a pressing event; and the first terminal displays, on the display screen of the first terminal, the animation of the object moving with the second cursor according to the moving event, the pressing event and the drag data.

[0044] In another possible implementation, after the first terminal determines that the second cursor slides out of the edge of the display screen of the first terminal, the method can further include: the first terminal displays an invisible window, the transparency of the invisible window being greater than a threshold value, and the invisible window being used to initiate the drag event.

[0045] In another possible implementation manner, the long-press operation includes a press operation and a lift operation; the first terminal sends, to the second terminal, data of the long-press operation input by the user using the input device of the first terminal, including: after the user performs the press operation using the input device of the first terminal, the first terminal intercepts a press event, and the first terminal sends, to the second terminal, fourth operation parameters included in the press event; after the user performs the lift operation using the input device of the first terminal after performing the press operation for a period of time, the first terminal intercepts a lift event, and the first terminal sends, to the second terminal, fifth operation parameters included in the lift event; wherein the fourth operation parameters and the fifth operation parameters are respectively used by the second terminal to simulate the press event and the lift event, and are further used to trigger the windowing of the interface of the application.

[0046] In a third aspect, the present application provides a cross-device object dragging device, which can be applied to a second terminal connected with a first terminal, and can include: a display unit configured to display a first cursor on an object displayed on the second terminal; a receiving unit configured to receive a dragging operation input by a user using an input device of the first terminal, the dragging operation being used to initiate the dragging of the object; the display unit is further configured to display an animation of the object moving with the first cursor on a display screen of the second terminal in response to the dragging operation; and a sending unit configured to send, to the first terminal, dragging data when it is determined that the object is dragged out of an edge of the display screen of the second terminal.

[0047] In a possible implementation manner, the dragging data can be used by the first terminal to display an animation of the object moving with a second cursor on a display screen of the first terminal.

[0048] In another possible implementation manner, the object can be an icon of an application, or the object can be a window including an interface of the application; when the object is the icon of the application, the dragging data includes the icon of the application; and when the object is the window, the dragging data includes the interface of the application.

[0049] In another possible implementation manner, the receiving unit is further configured to receive an indication message from the first terminal, the indication message being sent by the first terminal after the user inputs a drag release operation using the input device of the first terminal, and being used to indicate that the dragging of the object is ended; and the sending unit is further configured to send, to the first terminal, data used by the first terminal to display the interface of the application on a display screen of the first terminal.

[0050] In another possible implementation manner, when the object is the icon of the application, the device further includes a creating unit; the creating unit is configured to create a dragging object according to the icon of the application, the dragging object including an icon same as the icon of the application; and the display unit is specifically configured to display an animation of the dragging object moving with the first cursor in response to the dragging operation.

[0051] In a possible implementation, when the object is a window, the display unit is further configured to display an interface of the application; and the receiving unit is further configured to receive a long-press operation input by a user using an input device of the first terminal when the first cursor is displayed on the status bar of the second terminal, so as to trigger the interface windowing of the application; and the display unit is further configured to display the window in response to the long-press operation, wherein the window specifically includes a screenshot of the interface of the application, and the drag data specifically includes the screenshot of the interface of the application.

[0052] In a possible implementation, the display unit is specifically configured to display an animation of the shadow of the object moving along with the first cursor on the display screen of the second terminal.

[0053] In a possible implementation, the object can be text, a file or a folder; the drag data includes a bitmap of the shadow and content of the drag event; when the object is text, the content of the drag event includes the text; and when the object is a file or a folder, the content of the drag event is a file path.

[0054] In a possible implementation, the apparatus can further include an obtaining unit configured to obtain and cache the drag data.

[0055] In a possible implementation, the receiving unit is further configured to receive a request message from the first terminal, the request message being used to request the drag data; and the sending unit is specifically configured to send the cached drag data to the first terminal.

[0056] In a possible implementation, the receiving unit is further configured to receive first shuttle state information from the first terminal, the first shuttle state information being used to indicate the start of the shuttle; and the display unit is further configured to display the first cursor on the second terminal in response to the first shuttle state information.

[0057] In a possible implementation, the receiving unit is further configured to receive an operation of moving the first cursor input by a user using an input device of the first terminal; and the display unit is further configured to display an animation of the first cursor moving on the display screen of the second terminal in response to the operation of moving the first cursor, until the first cursor moves to the object.

[0058] In another possible implementation, the drag operation includes a pressing operation and a moving operation; the receiving unit is specifically configured to receive a first operation parameter from the first terminal, the first operation parameter being an operation parameter contained in a pressing event received by the first terminal after the user performs the pressing operation using the input device of the first terminal; receive a second operation parameter from the first terminal, the second operation parameter being an operation parameter contained in a moving event received by the first terminal after the user performs the moving operation using the input device of the first terminal; the apparatus can further include: a simulation unit configured to simulate the pressing event according to the first operation parameter, and simulate the moving event according to the second operation parameter, the pressing event and the moving event being used to instruct to initiate the drag operation on the object; and a display unit configured to display an animation of the object moving with the first cursor in response to the pressing event and the moving event.

[0059] In another possible implementation, the receiving unit is specifically configured to receive a third operation parameter from the first terminal, the third operation parameter being an operation parameter contained in a moving event received by the first terminal after the user performs the moving operation using the input device of the first terminal; the apparatus can further include: a simulation unit configured to simulate the moving event according to the third operation parameter, the moving event being used to instruct to move the first cursor.

[0060] In another possible implementation, the sending unit is further configured to send second shuttle state information to the first terminal, the second shuttle state information being used to indicate that the shuttle ends.

[0061] In another possible implementation, the long pressing operation includes a pressing operation and a lifting operation; the receiving unit is specifically configured to receive a fourth operation parameter from the first terminal, the fourth operation parameter being an operation parameter contained in a pressing event received by the first terminal after the user performs the pressing operation using the input device of the first terminal when the first cursor is displayed on the status bar; receive a fifth operation parameter from the first terminal, the fifth operation parameter being an operation parameter contained in a lifting event received by the first terminal after the user performs the lifting operation using the input device of the first terminal after performing the pressing operation for a period of time; the apparatus can further include: a simulation unit configured to simulate the pressing event according to the fourth operation parameter, and simulate the lifting event according to the fifth operation parameter, the pressing event and the lifting event being used to trigger the interface windowing of the application; and a display unit configured to display the window in response to the pressing event and the lifting event.

[0062] In another possible implementation, the creating unit is further configured to create the virtual input device after the connection with the first terminal is successfully established; or, the receiving unit is further configured to receive a notification message from the first terminal, the notification message being used to indicate that the key-mouse sharing mode of the first terminal has been started, and the creating unit is further configured to create the virtual input device in response to the notification message; wherein the virtual input device is used by the second terminal to simulate the input event according to the operation parameter.

[0063] In a fourth aspect of the present application, a device for cross-device object dragging is provided. The device is applied to a first terminal, and the first terminal is connected with a second terminal. The device can include: a sending unit configured to send, to the second terminal, data of a dragging operation input by a user using an input device of the first terminal, the dragging operation being used to initiate a dragging operation on an object displayed by the second terminal; and a receiving unit configured to receive, from the second terminal, dragging data sent by the second terminal to the first terminal after the second terminal determines that the object is dragged out of an edge of a display screen of the second terminal during movement of the object along with a first cursor on the display screen of the second terminal.

[0064] In a possible implementation, the device can further include a display unit configured to display, on a display screen of the first terminal, an animation of movement of the object along with a second cursor according to the dragging data and the dragging operation.

[0065] In another possible implementation, the object can be an icon of an application, or the object can be a window including an interface of the application. When the object is the icon of the application, the dragging data includes the icon of the application. When the object is the window, the dragging data includes the interface of the application.

[0066] In another possible implementation, the receiving unit is further configured to receive a dragging release operation input by the user using the input device of the first terminal, and the display unit is further configured to display, on the display screen of the first terminal, the interface of the application in response to the dragging release operation.

[0067] In another possible implementation, the sending unit is further configured to send, to the second terminal, an indication message indicating that the dragging of the object is ended, the receiving unit is further configured to receive data from the second terminal, and the display unit is specifically configured to display, on the display screen of the first terminal, the interface of the application according to the data.

[0068] In another possible implementation, when the object is the window, the sending unit is further configured to send, to the second terminal, data of a long-press operation input by the user using the input device of the first terminal, the long-press operation being input by the user when the first cursor is displayed on a status bar of the second terminal and being used to trigger windowing of the interface of the application displayed by the second terminal, and the window specifically includes a screenshot of the interface of the application, and the dragging data specifically includes the screenshot of the interface of the application.

[0069] In another possible implementation, the display unit is specifically configured to display, on the display screen of the first terminal, an animation of movement of a shadow of the object along with the second cursor according to the dragging data and the dragging operation.

[0070] In another possible implementation, the object can be a text, a file or a folder; the drag data includes a bitmap of the drag event content and a shadow; and when the object is a text, the drag event content includes the text, and when the object is a file or a folder, the drag event content is a file path.

[0071] In another possible implementation, the apparatus can further include a determination unit configured to determine that the second cursor slides out of an edge of the display screen of the first terminal.

[0072] In another possible implementation, the receiving unit is further configured to receive an operation of moving the second cursor input by a user using an input device of the first terminal; and the display unit is further configured to display an animation of movement of the second cursor on the display screen of the first terminal in response to the operation of moving the second cursor.

[0073] In another possible implementation, the sending unit is further configured to send, to the second terminal, first shuttle state information, the first shuttle state information being used to indicate a start of the shuttle.

[0074] In another possible implementation, the sending unit is further configured to send, to the second terminal, data of an operation of moving the first cursor input by a user using an input device of the first terminal, the data being used by the second terminal to display an animation of movement of the first cursor on a display screen of the second terminal.

[0075] In another possible implementation, the drag operation includes a press operation and a movement operation; the apparatus can further include an intercepting unit configured to intercept a press event after the user performs the press operation using the input device of the first terminal; and the sending unit is specifically configured to send, to the second terminal, first operation parameters included in the press event; and the intercepting unit is further configured to intercept a movement event after the user performs the movement operation using the input device of the first terminal; and the sending unit is specifically configured to send, to the second terminal, second operation parameters included in the movement event; wherein the first operation parameters and the second operation parameters are respectively used by the second terminal to simulate the press event and the movement event, and are further used to indicate initiation of the drag on the object.

[0076] In another possible implementation, the intercepting unit is configured to intercept a movement event after the user performs the movement operation using the input device of the first terminal; and the sending unit is specifically configured to send, to the second terminal, third operation parameters included in the movement event, the third operation parameters being used by the second terminal to simulate the movement event, and are further used to indicate movement of the first cursor.

[0077] In another possible implementation, the receiving unit is further configured to receive second shuttle state information from the second terminal, the second shuttle state information being used to indicate an end of the shuttle.

[0078] In a possible implementation, the sending unit is further configured to send a request message to the second terminal, the request message being used to request the drag data.

[0079] In a possible implementation, the apparatus further includes an intercepting unit and an input unit. The intercepting unit is configured to intercept the input event. The input unit is configured to receive a movement event after the user performs the movement operation using the input device of the first terminal. The input unit is further configured to generate a press event. The display unit is specifically configured to display an animation of the object moving with the second cursor on the display screen of the first terminal according to the movement event, the press event, and the drag data.

[0080] In a possible implementation, the display unit is further configured to display an invisible window, the invisible window having a transparency greater than a threshold value, and the invisible window being used to initiate the drag event.

[0081] In a possible implementation, the long press operation includes a press operation and a lift operation. The intercepting unit is configured to intercept a press event after the user performs the press operation using the input device of the first terminal. The sending unit is specifically configured to send a fourth operation parameter included in the press event to the second terminal. The intercepting unit is further configured to intercept a lift event after the user performs the lift operation using the input device of the first terminal after performing the press operation for a period of time. The sending unit is further configured to send a fifth operation parameter included in the lift event to the second terminal. The fourth operation parameter and the fifth operation parameter are respectively used to simulate the press event and the lift event of the second terminal, and further used to trigger the interface windowing of the application.

[0082] In a fifth aspect, the present application provides a device for cross-device object dragging, including a processor, a memory for storing processor-executable instructions, wherein the processor is configured to execute the instructions to implement the method in the first aspect or any possible implementation of the first aspect, or implement the method in the second aspect or any possible implementation of the second aspect.

[0083] In a sixth aspect, the present application provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the method in the first aspect or any possible implementation of the first aspect, or implement the method in the second aspect or any possible implementation of the second aspect.

[0084] A seventh aspect of this application provides a terminal including a display screen, one or more processors, and a memory; the display screen, processors, and memory are coupled; the memory is used to store computer program code, the computer program code including computer instructions, which, when executed by the terminal, cause the terminal to perform a method as described in any one of the first aspect or any possible implementation thereof, or cause the terminal to perform a method as described in any one of the second aspect or any possible implementation thereof.

[0085] An eighth aspect of this application provides a computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is executed in a terminal, a processor in the terminal executes the method of any one of the first aspect or any possible implementation thereof, or executes the method of any one of the second aspect or any possible implementation thereof.

[0086] A ninth aspect of this application provides a cross-device object drag-and-drop system, which may include a first terminal and a second terminal, the first terminal being connected to the second terminal.

[0087] The second terminal displays a first cursor on the object displayed on the second terminal; the first terminal sends drag operation data input by the user using the input device of the first terminal to the second terminal; the second terminal receives the drag operation input by the user using the input device of the first terminal, which is used to initiate a drag on an object; in response to the drag operation, the second terminal displays an animation of the object moving with the first cursor on the display screen of the second terminal; when the second terminal determines that the object has been dragged off the edge of the display screen of the second terminal, it sends drag data to the first terminal. The first terminal receives the drag data from the second terminal.

[0088] In one possible implementation, the first terminal displays an animation of an object moving with a second cursor on its screen, based on the drag data and drag operation.

[0089] In another possible implementation, after determining that the second cursor has slid off the edge of the display screen of the first terminal, the first terminal sends data of the drag operation input by the user using the input device of the first terminal to the second terminal.

[0090] In another possible implementation, after the first terminal determines that the second cursor has slid out of the edge of the first terminal's display screen, the first terminal sends first shuttle status information to the second terminal, which is used to indicate that the shuttle has started.

[0091] In another possible implementation manner, the second terminal further sends second shuttle state information to the first terminal after determining that the object is dragged out of the edge of the display screen of the second terminal, and the second shuttle state information is used to indicate that the shuttle ends.

[0092] It should be noted that the object dragged out of the edge of the display screen of the terminal (such as the second terminal) in the embodiment can be that a partial region of the object is dragged out (or overflowed) from the display screen of the terminal, or that the entire region of the object is dragged out (or overflowed) from the display screen of the terminal, or that the object slides out of the edge of the display screen of the terminal when the cursor moves on the display screen of the terminal, and the embodiment does not make a specific limitation here.

[0093] It can be understood that the beneficial effects of the cross-device object dragging apparatus provided in the third aspect and any possible implementation manner thereof, the cross-device object dragging apparatus provided in the fourth aspect and any possible implementation manner thereof, the cross-device object dragging apparatus provided in the fifth aspect, the computer readable storage medium provided in the sixth aspect, the terminal provided in the seventh aspect, the computer program product provided in the eighth aspect, and the cross-device object dragging system provided in the ninth aspect can refer to the beneficial effects in the first aspect or the second aspect and any possible implementation manner thereof, and details are not described herein again. BRIEF DESCRIPTION OF DRAWINGS

[0094] Figure 1 A simplified schematic diagram of a system architecture provided in the embodiment of the application;

[0095] Figure 2A A structural schematic diagram of a mobile phone provided in the embodiment of the application;

[0096] Figure 2B A component schematic diagram of a software architecture provided in the embodiment of the application;

[0097] Figure 3 A flow schematic diagram of a cross-device object dragging method provided in the embodiment of the application;

[0098] Figure 4 A schematic diagram of a coordinate system on a display screen provided in the embodiment of the application;

[0099] Figure 5A A cross-device object dragging interface schematic diagram provided in the embodiment of the application;

[0100] Figure 5B Another cross-device object dragging interface schematic diagram provided in the embodiment of the application;

[0101] Figure 6 Still another cross-device object dragging interface schematic diagram provided in the embodiment of the application;

[0102] Figure 7 Another cross-device object dragging interface schematic diagram provided by an embodiment of the present application;

[0103] Figure 8 Another cross-device object dragging interface schematic diagram provided by an embodiment of the present application;

[0104] Figure 9 Another cross-device object dragging method flow schematic diagram provided by an embodiment of the present application;

[0105] Figure 10 Another cross-device object dragging interface schematic diagram provided by an embodiment of the present application;

[0106] Figure 11 Another cross-device object dragging interface schematic diagram provided by an embodiment of the present application;

[0107] Figure 12 Another cross-device object dragging interface schematic diagram provided by an embodiment of the present application;

[0108] Figure 13 Another cross-device object dragging interface schematic diagram provided by an embodiment of the present application;

[0109] Figure 14 Another cross-device object dragging interface schematic diagram provided by an embodiment of the present application;

[0110] Figure 15 Another cross-device object dragging method flow schematic diagram provided by an embodiment of the present application;

[0111] Figure 16 Another cross-device object dragging interface schematic diagram provided by an embodiment of the present application;

[0112] Figure 17 A windows-side drag event data structure schematic diagram provided by an embodiment of the present application;

[0113] Figure 18 An Android-side drag event data structure schematic diagram provided by an embodiment of the present application;

[0114] Figure 19 Another cross-device object dragging interface schematic diagram provided by an embodiment of the present application;

[0115] Figure 20 Another cross-device object dragging method flow schematic diagram provided by an embodiment of the present application;

[0116] Figure 21 Another cross-device object dragging interface schematic diagram provided by an embodiment of the present application;

[0117] Figure 22 Another cross-device object dragging interface provided by the embodiment of the present application;

[0118] Figure 23 A cross-device object dragging device provided by the embodiment of the present application;

[0119] Figure 24 Another cross-device object dragging device provided by the embodiment of the present application;

[0120] Figure 25 A chip system provided by the embodiment of the present application. DETAILED DESCRIPTION

[0121] Hereinafter, the terms "first" and "second" are only used for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the embodiment of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0122] The embodiment of the present application provides a cross-device object dragging method and device, which can be applied to a scenario in which multiple terminals are used cooperatively. In the scenario, the same input device (such as a mouse or a touchpad) is used to control all terminals by means of a keyboard and mouse sharing technology. For example, the objects such as application icons or application interfaces can be dragged across devices. For another example, the content (or objects) such as text or files can be transmitted among multiple terminals participating in cooperative use by means of dragging, and the user is allowed to process the transmitted content using the terminals. The method provided by the embodiment can increase the interaction between terminals when multiple terminals are used cooperatively, and improve the user experience.

[0123] The implementation of the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0124] Reference is made to Figure 1 A simplified schematic diagram of a system architecture to which the above method can be applied is provided by the embodiment of the present application. The system architecture can be a cross-device object dragging system in the embodiment. As Figure 1 indicated, the system architecture can at least include a first terminal 101 and a second terminal 102.

[0125] The first terminal 101 is connected with an input device 101-1 (as Figure 1 indicated), or includes the input device 101-1 (as Figure 1The input device 101-1 can be a mouse, a touchpad, a touch screen, etc. Figure 1 In this embodiment, the input device 101-1 is taken as an example of a mouse.

[0126] In this embodiment, the first terminal 101 and the second terminal 102 can establish a connection in a wired or wireless manner. Based on the established connection, the first terminal 101 and the second terminal 102 can be used together. In this embodiment, the wireless communication protocol used when the first terminal 101 and the second terminal 102 establish a connection in a wireless manner can be a wireless fidelity (Wi-Fi) protocol, a Bluetooth protocol, a ZigBee protocol, a Near Field Communication (NFC) protocol, etc., and can also be various cellular network protocols, which are not specifically limited here.

[0127] After the first terminal 101 and the second terminal 102 are connected, the user can use a set of input devices, such as the input device 101-1 described above, to control both the first terminal 101 and the second terminal 102 by using the mouse and keyboard sharing technology. That is, the user can not only use the input device 101-1 of the first terminal 101 to control the first terminal 101, but the first terminal 101 can also share its input device 101-1 with the second terminal 102 to enable the user to control the second terminal 102.

[0128] For example, one or more applications are installed in the second terminal 102. The second terminal 102 can display icons of the corresponding applications on the display screen 102-1 of the second terminal 102. In this embodiment, after the first terminal 101 and the second terminal 102 are connected, the user can use the input device 101-1 described above to drag the icons of the applications from the display screen 102-1 of the second terminal 102 to the display screen 101-2 of the first terminal 101 by using the mouse and keyboard sharing technology.

[0129] For another example, the interface of an application is displayed on the display screen 102-1 of the second terminal 102. In this embodiment, after the first terminal 101 and the second terminal 102 are connected, the user can use the input device 101-1 described above to drag the interface of the application from the display screen 102-1 of the second terminal 102 to the display screen 101-2 of the first terminal 101 by using the mouse and keyboard sharing technology.

[0130] For another example, the input device 101-1 is a mouse. After the first terminal 101 is connected with the second terminal 102, without starting the screen projection, the user can use the mouse to drag the content of the first terminal 101, such as text or file, to the second terminal 102. The user can also use the mouse to drag the content of the second terminal 102, such as text or file, to the first terminal 101.

[0131] It should be noted that, in the embodiment of the present application, without starting the screen projection, the content can be dragged between two terminals, and also can be dragged between three or more terminals. For example, when the second terminal 102 is connected with another device, such as a third terminal, in the embodiment of the present application, without starting the screen projection, after the user drags the content, such as text or file, from one terminal to another terminal, the user can continue to drag the content to the third terminal. For example, continuing the above example, without starting the screen projection, after the user uses the mouse to drag the content, such as text or file, of the first terminal 101 to the second terminal 102, the user can continue to drag the content to the third terminal. After the user releases the mouse, the content dragging is completed.

[0132] It should be noted that, in the embodiment of the present application, the terminal, such as the first terminal 101, the second terminal 102 and the third terminal, can be a mobile phone, a tablet computer, a handheld computer, a PC, a cellular phone, a personal digital assistant (PDA), a wearable device (such as a smart watch), a smart home device (such as a television), an in-vehicle computer, a game console, an augmented reality (AR) device, a virtual reality (VR) device, etc. The specific form of the terminal is not limited in the embodiment. For example, Figure 1 In the embodiment, the first terminal 101 is a PC and the second terminal 102 is a mobile phone.

[0133] In the embodiment, the terminal is a mobile phone. Please refer to Figure 2A FIG. 1 is a structural schematic diagram of a mobile phone provided in the embodiment of the present application.

[0134] As shown in FIG. 1, the mobile phone includes a processor 101, a memory 102, a display 103, a touch screen 104, a microphone 105, a speaker 106, a camera 107, a sensor 108, a communication interface 109, and a power supply 110. Figure 2AAs shown, the mobile phone can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0135] The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0136] It can be understood that the structure shown in the embodiment does not constitute a specific limitation on the mobile phone. In other embodiments, the mobile phone can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0137] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or can be integrated in one or more processors.

[0138] The controller can be the nerve center and command center of the mobile phone. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching instructions and executing instructions.

[0139] The processor 110 can also include memory that stores instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The cache memory can hold instructions or data that the processor 110 has recently used or that are likely to be used. If the processor 110 needs to use the instructions or data again, it can be retrieved directly from the cache memory. This avoids repeated accesses to the main memory, reducing the latency of the processor 110 and improving the efficiency of the system.

[0140] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface, among others.

[0141] The charging management module 140 is configured to receive charging input from a charger. The charging management module 140 can also supply power to the phone through the power management module 141 while charging the battery 142. The power management module 141 is configured to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 can also receive input from the battery 142 to supply power to the phone.

[0142] The wireless communication functions of the phone can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, and a baseband processor, among others.

[0143] The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals. Each antenna in the phone can be configured to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0144] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc. on a mobile phone. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and perform filtering, amplification, etc. on the received electromagnetic waves, and transfer the processed signals to the modem processor for demodulation. The mobile communication module 150 can also amplify signals modulated by the modem processor, and radiate the amplified signals as electromagnetic waves through the antenna 1. In some embodiments, at least part of the functions of the mobile communication module 150 can be provided in the processor 110. In some embodiments, at least part of the functions of the mobile communication module 150 can be provided in the same device as at least part of the processor 110.

[0145] The modem processor can include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is configured to demodulate a received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the microphone 170B, etc.), or displays an image or a video through the display screen 194. In some embodiments, the modem processor can be a separate device. In other embodiments, the modem processor can be independent of the processor 110, and provided in the same device as the mobile communication module 150 or other functional modules.

[0146] The wireless communication module 160 can provide a solution for wireless communication including wireless local area networks (WLAN) (e.g., Wi-Fi networks), Bluetooth (BT), global navigation satellite systems (GNSS), frequency modulation (FM), NFC, infrared (IR) technology, etc. on a mobile phone. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering on the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 can also receive signals to be transmitted from the processor 110, perform frequency modulation and amplification, and radiate the processed signals as electromagnetic waves through the antenna 2.

[0147] In some embodiments, the antennas 1 and the mobile communication module 150 are coupled, and the antennas 2 and the wireless communication module 160 are coupled, so that the mobile phone can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS can include global positioning system (GPS), global navigation satellite system (GLONASS), beidu navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0148] The mobile phone implements display functions through a GPU, the display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, connected with the display screen 194 and the application processor. The processor 110 can include one or more GPUs, which execute program instructions to generate or change display information.

[0149] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a Micro Led, a Micro-oLed, a quantum dot light emitting diode (QLED), or the like. In some embodiments, the mobile phone can include one or N display screens 194, where N is a positive integer greater than 1.

[0150] The mobile phone can implement the photographing function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, and the like. In some embodiments, the mobile phone can include one or N cameras 193, where N is a positive integer greater than 1.

[0151] The external memory interface 120 can be configured to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, the music, video, and the like are saved in the external memory card.

[0152] The internal memory 121 can be configured to store computer executable program codes including instructions. The processor 110 executes various function applications and data processing of the mobile phone by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, and the like), and the like. The data storage area can store data created during the use of the mobile phone (such as audio data, a phone book, and the like), and the like. In addition, the internal memory 121 can include a high-speed random access memory, and can further include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.

[0153] The mobile phone can implement the audio function through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, and the like. For example, music playing, recording, and the like.

[0154] The pressure sensor 180A is configured to sense a pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. The pressure sensor 180A can be of various types, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, etc. When a touch operation is performed on the display screen 194, the phone can determine the intensity of the touch operation based on the pressure sensor 180A. The phone can also determine the location of the touch based on the detection signal of the pressure sensor 180A.

[0155] The gyroscope sensor 180B can be configured to determine the motion posture of the phone. The barometric pressure sensor 180C can be configured to measure the barometric pressure. The magnetic sensor 180D can include a Hall sensor. The phone can use the magnetic sensor 180D to detect the opening and closing of a flip cover. The acceleration sensor 180E can be configured to detect the acceleration of the phone in various directions (typically three axes). The distance sensor 180F can be configured to measure the distance. The phone can use the proximity light sensor 180G to detect that the user is holding the phone close to the ear for a call, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used for automatic unlocking and locking of the phone in a cover mode or a pocket mode. The ambient light sensor 180L can be configured to sense the ambient light intensity. The fingerprint sensor 180H can be configured to collect a fingerprint. The phone can use the collected fingerprint characteristics to realize fingerprint unlocking, access application locking, fingerprint photographing, fingerprint answering a call, etc. The temperature sensor 180J can be configured to detect the temperature.

[0156] The touch sensor 180K, also referred to as a "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 can form a touch screen, also referred to as a "touch panel". The touch sensor 180K can be configured to detect a touch operation performed thereon or in the vicinity thereof. The touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the phone, which is different from the location of the display screen 194.

[0157] The bone conduction sensor 180M can be configured to obtain a vibration signal. The keys 190 can include a power key, a volume key, etc. The keys 190 can be mechanical keys. They can also be touch keys. The motor 191 can be configured to generate a vibration prompt. The motor 191 can be used for incoming call vibration prompt, and can also be used for touch vibration feedback. The indicator 192 can be an indicator light, which can be used to indicate the charging state, the power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0158] The SIM card interface 195 is configured to connect a SIM card. The SIM card can be connected to or disconnected from the mobile phone by being inserted into or pulled out of the SIM card interface 195. The mobile phone can support one or N SIM card interfaces, where N is a positive integer greater than 1. The mobile phone can interact with a network through the SIM card to implement functions such as call and data communication. In some embodiments, the mobile phone uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the mobile phone and cannot be disconnected from the mobile phone. The methods in the following embodiments can be implemented in a mobile phone having the hardware structure described above.

[0159] In combination Figure 1 , the software system of the first terminal 101 is a Windows system, and the software system of the second terminal 102 is an Android system. The software structure of the first terminal 101 and the second terminal 102 is described by way of example. Please refer to Figure 2B , a component diagram of a software architecture provided by the embodiments of the present application.

[0160] As shown in Figure 2B , the software architecture of the first terminal 101 can include an application layer and a Windows system (Windows shell). In some embodiments, the application layer can include various applications installed on the first terminal 101. The applications of the application layer can directly interact with the Windows system. By way of example, the application layer can further include a keyboard and mouse module, a drag management module, and a transmission management module. Further, the application layer can further include a window management module.

[0161] The software system of the second terminal 102 can use a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. By way of example, the software system of the second terminal 102 is a layered architecture. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, as shown in Figure 2BAs shown, the second terminal 102 can include an application layer and a framework layer (FWK). The application layer can include a series of application packages. For example, the application packages can include settings, calculator, camera, short message, music player, and the like applications. The applications included in the application layer can be system applications of the second terminal 102, or third-party applications, and the embodiments of the present application do not make specific limitations here. The application layer can further include a transmission management module and a drag management module. The application layer can further include a launcher. The application layer can further include a system UI. The framework layer is mainly responsible for providing application programming interfaces (APIs) and programming frameworks for the applications of the application layer. In some embodiments, the framework layer can include a window manager (or a window management service). Of course, the second terminal 102 can further include other layers, such as a kernel layer (not shown in the figure), and the like. The kernel layer is a layer between hardware and software. The kernel layer can at least include display drivers, camera drivers, audio drivers, sensor drivers, and the like. Figure 2B

[0162] In the scenario of multi-terminal cooperation, in order to improve the use efficiency, the user can use the same input device to control the multiple terminals. In the embodiments, combined with Figure 1 , taking the input device 101-1 of the first terminal 101 as a mouse as an example, after the first terminal 101 and the second terminal 102 establish a connection, based on the above software architecture and with the aid of the keyboard and mouse sharing technology, the user can use the input device 101-1 of the first terminal 101 to control the first terminal 101 and the second terminal 102. For example, in some embodiments, the user can use the input device 101-1 of the first terminal 101 to realize the drag of the icon of the application or the interface of the application displayed by the second terminal 102 from the second terminal 102 to the first terminal 101. For another example, in some other embodiments, without starting the screen projection, the user can use the input device 201-1 of the first terminal 201 to realize the drag of the file or the text and the like content of the first terminal 201 to the second terminal 202. The file or the text and the like content of the second terminal 202 can also be dragged to the first terminal 201. That is, without starting the screen projection, the user can use the input device 201-1 of the first terminal 201 to realize the bidirectional drag of the file or the text and the like content in the application between the first terminal 201 and the second terminal 202 through the drag.

[0163] Among them, the keyboard and mouse sharing technology can refer to a technology of using the input device (such as a mouse, a touchpad) of one terminal to control other terminals.

[0164] ​It should be noted that in this embodiment, the drag management module can also be referred to as a drag service module. In addition, in the above embodiment, it can be seen that the first terminal 101 and the second terminal 102 each include a transmission management module, and the communication between the first terminal 101 and the second terminal 102 can be realized through the transmission management module. In some other embodiments, the drag management module can also have the function of communicating with other terminals, that is, the first terminal 101 and the second terminal 102 can also not include the transmission management module, and the communication between them can be realized through the drag management module, and the present embodiment does not make specific limitations here. For the convenience of description, the following embodiments are described by taking the communication between the first terminal 101 and the second terminal 102 realized through the transmission management module as an example.

[0165] In some embodiments, the following is combined with Figure 1 and Figure 2B Take the first terminal 101 as a PC and the second terminal 102 as a mobile phone, and the input device 101-1 as a mouse as an example, and the method provided by the present embodiment is described in detail in combination with the drawings.

[0166] Figure 3 A flowchart of a cross-device object dragging method provided by an embodiment of the present application. In this embodiment, the implementation process of the user using the mouse shared by the PC to drag the icon of the application displayed by the mobile phone (the icon of the application is the object to be dragged) from the mobile phone to the PC is introduced. As shown in Figure 3 The method can include the following S301-S313.

[0167] S301, the mobile phone and the PC establish a connection.

[0168] In some embodiments, the mobile phone and the PC can establish a connection in a wired manner. For example, the mobile phone and the PC can establish a wired connection through a data line.

[0169] In some other embodiments, the mobile phone and the PC can establish a connection in a wireless manner. In this case, two requirements exist for the terminals to establish a connection in a wireless manner. One is that the terminals need to know the connection information of the opposite terminal, and the other is that the terminals need to have a transmission capability. The connection information can be the device identifier of the terminal, such as an Internet Protocol (IP) address, a port number, or a terminal login account. The terminal login account can be an account provided by an operator for a user, such as a Huawei account. The terminal login account can also be an application account, such as a WeChat account or a Youku account. The transmission capability of the terminal can be a near field communication capability or a long distance communication capability. That is, the wireless communication protocol used by the terminals, such as the mobile phone and the PC, to establish a connection can be a near field communication protocol, such as a Wi-Fi protocol, a Bluetooth protocol, or an NFC protocol, or can be a cellular network protocol. Taking the mobile phone and the PC as an example, the user can use the mobile phone to touch the NFC tag of the PC, and the mobile phone reads the connection information saved in the NFC tag, such as the IP address of the PC. Then, the mobile phone can establish a connection with the PC according to the IP address of the PC using the NFC protocol. For another example, the mobile phone and the PC both open the Bluetooth function and the Wi-Fi function. The PC can broadcast a Bluetooth signal to discover the surrounding terminals, such as the PC can display a discovered device list, which can include the identifier of the mobile phone discovered by the PC. In addition, the PC and the discovered device can exchange connection information, such as an IP address, during the device discovery process. Then, after the PC receives the operation of the user selecting the identifier of the mobile phone in the displayed device list, the PC can establish a connection with the mobile phone using the Wi-Fi protocol according to the IP address of the mobile phone. For another example, the mobile phone and the PC are both connected to a cellular network, and the mobile phone and the PC are logged in to the same Huawei account. The mobile phone and the PC can establish a connection based on the cellular network according to the Huawei account.

[0170] After the mobile phone and the PC successfully establish a connection, the two can be used cooperatively. In order to improve the efficiency of cooperative use, the user can use a set of input devices, such as the mouse of the PC, to control both the PC and the mobile phone.

[0171] As an exemplary implementation, in the case where the PC is in a mouse and keyboard sharing mode, a set of input devices can be used to control both the PC and the mobile phone.

[0172] For example, in some embodiments, after the other terminal and the PC successfully establish a connection, the PC can display a pop-up window to ask the user whether to open the mouse and keyboard sharing mode. If the operation of the user selecting to open the mouse and keyboard sharing mode is received, the PC can open the mouse and keyboard sharing mode.

[0173] PC can notify all terminals which have established connection with the PC that the keyboard and mouse sharing mode has been opened. If the PC has established connection with the mobile phone, the PC can notify the mobile phone that the keyboard and mouse sharing mode has been opened. After receiving the notification (referred to as a notification message), the mobile phone can create a virtual input device which has the same function as a conventional input device such as a mouse or a touchpad and can be used to simulate corresponding input events on the mobile phone. For example, taking the input device as a mouse, the virtual input device created by the mobile phone has the same function as a conventional mouse and can be regarded as a mouse shared by the PC to the mobile phone and can be used to simulate mouse events on the mobile phone to realize the control of the mobile phone by the mouse of the PC. Taking the Android system as an example, the mobile phone can use the uinput capability of Linux to create the virtual input device. The uinput is a kernel layer module which can simulate an input device. By writing into the / dev / uinput (or / dev / input / uinput) device, a process can create a virtual input device with specific functions. Once the virtual input device is created, it can simulate corresponding events. Similarly, other terminals which have established connection with the PC can also create virtual input devices according to the received notification message. It should be noted that if the operating system of the terminal receiving the notification message is the Android system, the uinput capability of Linux can be used to create the virtual input device or the human interface device (HID) protocol can be used to create the virtual input device. If the operating system of the terminal receiving the notification message is the IOS system or other operating systems such as the windows system, the HID protocol can be used to create the virtual input device. In addition, the above embodiment is described taking the terminal connected with the PC receiving the notification message for notifying that the keyboard and mouse sharing mode of the PC has been opened and then creating the virtual input device as an example. In some other embodiments, after the terminal connected with the PC receives the above notification message, a pop-up window can be displayed to ask the user whether to use the input device of the PC to control the terminal. If the user selects to use the input device of the PC to control the terminal, the virtual input device is created, otherwise, the virtual input device is not created.

[0174] For example, in some embodiments, after the mobile phone establishes connection with the PC, the PC automatically opens the keyboard and mouse sharing mode, and the user does not need to manually open the keyboard and mouse sharing mode. In other embodiments, after the mobile phone establishes connection with the PC, the PC automatically creates a virtual input device, and the PC does not need to send a notification message. Alternatively, after the mobile phone establishes connection with the PC, a pop-up window can be displayed to ask the user whether the user wants to use the input device of the PC to control the mobile phone. If the user selects to use the input device of the PC to control the mobile phone, the PC automatically creates a virtual input device. Otherwise, the PC does not create a virtual input device.

[0175] In addition, in combination with Figure 1 Since the mouse is an input device of the PC, after the mobile phone establishes connection with the PC, the mouse is generally temporarily responsive to the operation of the PC, or the user can temporarily use the mouse to control the PC. In this embodiment, after the PC opens the keyboard and mouse sharing mode, the PC can also trigger the mobile phone to respond to the operation of the mouse when it is determined that the mouse switching condition is met. That is, after the mouse switching condition is met, the user can use the mouse to control the mobile phone.

[0176] For example, the mouse switching condition can be that the mouse pointer corresponding to the mouse displayed on the display screen of the PC slides over the edge of the display screen of the PC. That is, the user can move the mouse to make the mouse pointer corresponding to the mouse displayed on the display screen of the PC slide over the edge of the display screen of the PC to trigger the mobile phone to respond to the operation of the mouse.

[0177] As an exemplary implementation, after starting the mouse and keyboard sharing mode, the PC can start input monitoring and mount a hook. The input monitoring can be used to monitor the relative displacement and coordinate position of the mouse pointer. The mounted hook can be used to intercept (or shield) the corresponding input event, such as a mouse event, after the mouse teleportation is started, so that the mouse event is not transmitted to the windows system of the PC after being received by the mouse and keyboard module of the PC. The mounted hook can also be used to capture the intercepted input event, such as the parameters in the mouse event, after the mouse teleportation is started. The PC can monitor the relative displacement and coordinate position of the mouse pointer by using the input monitoring, and determine whether the mouse teleportation condition is met according to the monitored data. After determining that the mouse teleportation condition is met, the mounted hook intercepts the mouse event, captures the operation parameters in the mouse event, and sends the captured operation parameters to the other terminal connected with the PC, which creates a virtual input device, so that the other terminal simulates the corresponding input event, such as a mouse event, by using the created virtual input device, and then responds to it. That is, the other terminal connected with the PC, which creates a virtual input device, responds to the operation of the mouse.

[0178] Of course, the interception of the input event and the capture of the operation parameters therein can also be implemented by other means, such as registering RAWINPUT in the PC. Alternatively, the interception of the input event and the capture of the operation parameters therein can also be implemented by different means respectively. For example, taking the input device as a mouse, after starting the mouse and keyboard sharing mode, the PC can mount a hook and register RAWINPUT. After the mouse teleportation is started, the mounted hook can be used to intercept a mouse event (or shield a mouse event), and the registered RAWINPUT can be used to capture the operation parameters in the intercepted mouse event. The specific implementation of the interception of the mouse event and the capture of the operation parameters therein in this embodiment is not limited. In order to facilitate description, the interception of the input event and the capture of the operation parameters therein in the following embodiments are introduced by way of example.

[0179] For example, taking the user wanting to drag the icon of the application displayed by the mobile phone from the mobile phone to the PC as an example. The process of triggering the other terminal connected with the PC, which creates a virtual input device, to respond to the operation of the mouse can include the following S302-S304.

[0180] S302, the PC receives a mouse movement event, and displays an animation of the movement of the mouse pointer on the display screen of the PC according to the mouse movement event.

[0181] It should be noted that the mouse pointer in the embodiment can also be referred to as a cursor. The cursor can be an image, which can be dynamic or static, and the style of the cursor can also be different in different cases. In the embodiment, the mouse pointer is taken as an example for description. The mouse pointer displayed on the PC display screen can be a second cursor in the application.

[0182] S303, the PC monitors the coordinate position of the mouse pointer on the PC display screen.

[0183] S304, the PC intercepts the mouse moving event according to the coordinate position of the mouse pointer on the PC display screen when determining that the mouse pointer slides out of the edge of the PC display screen, and sends the mouse operation parameter 1 contained in the mouse moving event to the mobile phone.

[0184] In the embodiment, after the mouse and keyboard sharing mode is started, when the user wants to use the mouse to control other terminals connected with the PC, such as the mobile phone, which has created a virtual input device, for example, wants to drag the icon of an application displayed on the mobile phone to the PC, the user can trigger the mouse shuttle condition by continuously moving the mouse in the same direction, so that the mouse pointer corresponding to the mouse pointer displayed on the PC display screen slides over (or slides out of) the edge of the PC display screen.

[0185] For example, the PC can determine the coordinate position of the mouse pointer on the PC display screen according to the initial position of the mouse pointer and the relative displacement, so as to determine whether the mouse pointer slides out of the edge of the PC display screen.

[0186] The initial position of the mouse pointer can be the coordinate position of the mouse pointer on the PC display screen when the mouse starts to move, or the coordinate position of the mouse pointer on the PC display screen before the mouse starts to move. The initial position of the mouse pointer can be the coordinate position in the coordinate system with the upper left corner of the PC display screen as the coordinate origin, the X axis pointing to the right edge of the PC display screen from the upper left corner, and the Y axis pointing to the lower edge of the PC display screen from the upper left corner. For example, the specific process of the PC to determine whether the mouse pointer slides out of the edge of the PC display screen can be: in combination with Figure 4 , the PC can establish a coordinate system with the initial coordinate position as the coordinate origin (such as the position o shown in Figure 4 ), the X axis pointing to the right edge of the PC display screen from the coordinate origin o, and the Y axis pointing to the upper edge of the PC display screen from the coordinate origin o. The PC can determine the coordinate values of the edges of the PC display screen in the coordinate system. The coordinate values of the edges of the PC display screen in the coordinate system can be determined according to the resolution of the PC display screen and the initial position of the mouse pointer. For example, as Figure 4As shown, in the coordinate system, the right edge of the PC display screen has a coordinate value of xl on the X axis, the left edge has a coordinate value of -x2 on the X axis, the upper edge has a coordinate value of yl on the Y axis, and the lower edge has a coordinate value of -y2 on the Y axis. After the mouse is moved, the mouse reports the relative displacement of the mouse pointer to the PC. The PC can calculate the coordinate position (x, y) of the mouse pointer on the PC display screen after the mouse is moved according to the relative displacement reported by the mouse. According to the coordinate position (x, y), the PC can determine whether the mouse pointer slides out of the edge of the PC display screen. For example, if the coordinate value x of the mouse pointer on the X axis is greater than xl, it can be determined that the mouse pointer slides out of the right edge of the PC display screen. If the coordinate value x of the mouse pointer on the X axis is less than -x2, it can be determined that the mouse pointer slides out of the left edge of the PC display screen. If the coordinate value y of the mouse pointer on the Y axis is greater than yl, it can be determined that the mouse pointer slides out of the upper edge of the PC display screen. If the coordinate value y of the mouse pointer on the Y axis is less than -y2, it can be determined that the mouse pointer slides out of the lower edge of the PC display screen.

[0187] After it is determined that the mouse pointer slides out of the edge of the PC display screen, i.e., the mouse shuttle condition is triggered, if the user continues to operate the input device of the PC, such as the mouse, the PC can send the operation data input by the user using the input device of the PC to the other terminal that creates the virtual input device. For example, after the user inputs an operation using the input device of the PC, the PC can intercept the received input event corresponding to the input event, such as a mouse movement event, a mouse press event, a mouse release event, etc., and transmit the operation parameters contained in the intercepted input event, such as mouse operation parameters, to the other terminal connected to the PC that creates the virtual input device, so as to respond to the operation of the input device of the PC, such as the mouse, by the terminal. Taking the input event as a mouse event as an example, the operation parameters (such as mouse operation parameters) contained in the input event (or mouse event) can include: a mouse key flag bit (used to indicate which operation of pressing, releasing, moving or scroll wheel scrolling is performed by the user on the mouse), coordinate information (used to indicate the X coordinate and Y coordinate of the mouse movement when the user moves the mouse), scroll wheel information (used to indicate the X axis distance and Y axis distance of the scroll wheel scrolling when the user operates the scroll wheel of the mouse), and key information (used to indicate which key of the left key, middle key or right key of the mouse is operated by the user).

[0188] In some embodiments, if there is only one device connected to the PC, such as the mobile phone mentioned above, and the mobile phone creates a virtual input device, the PC can transmit corresponding operation parameters to the mobile phone so as to respond to the operation of the input device of the PC, such as the mouse, by the mobile phone. If there are multiple devices connected to the PC, and some or all of the multiple devices create virtual input devices, the PC can display a list option on the display screen of the PC when it is determined that the mouse shuttle condition is triggered, and the list option includes the identification of the device connected to the PC that creates a virtual input device, such as the identification of the mobile phone mentioned above. The PC can determine the device that responds to the operation of the input device of the PC according to the selection of the user. For example, if the user selects the identification of the mobile phone, the PC can transmit corresponding operation parameters to the mobile phone so as to respond to the operation of the input device of the PC by the mobile phone. For example, after the mobile phone receives the corresponding operation parameters, the mobile phone can simulate corresponding input devices, such as mouse events, and make corresponding responses, i.e., respond to the operation of the input device of the PC by the mobile phone. In this embodiment, after the device connected to the PC completes the creation of the virtual input device, the device can send an indication message to the PC that the creation of the virtual input device is successful. The PC can obtain which devices connected to the PC successfully create virtual input devices according to the received indication message, and display the list option mentioned above based on this.

[0189] In some other embodiments, the shuttle relationship can be preconfigured. If there are multiple devices connected to the PC, and some or all of the multiple devices have created virtual input devices, the mouse can be determined to shuttle to which device, i.e., to determine which device to respond to the operation of the input device of the PC, according to the preconfigured shuttle relationship. For example, the multiple devices connected to the PC include the mobile phone described above, and the mobile phone has created a virtual input device, and the preconfigured shuttle relationship is that the mouse pointer slides out from the left side (or the left edge) of the display screen of the PC, then the mouse shuttles to the mobile phone. Then, when the user moves the mouse so that the mouse pointer slides over the left edge of the display screen of the PC, the PC can not only determine that the mouse has started to shuttle, but also determine that the mouse has shuttled to the mobile phone, i.e., the PC can send the corresponding operation parameters to the mobile phone so as to respond to the operation of the input device of the PC by the mobile phone. Of course, if there is only one device connected to the PC, and the device has created a virtual input device, the mouse can also be determined to shuttle to the device according to the preconfigured shuttle relationship. For example, the preconfigured shuttle relationship is that the mouse pointer slides out from the left edge of the display screen of the PC, then the mouse shuttles to the mobile phone. But when the user moves the mouse so that the mouse pointer slides over the right edge of the display screen of the PC, it can be determined that the mouse does not shuttle to the mobile phone. In some other embodiments, the mouse can be determined to shuttle to which device by recognizing the position of the device. For example, taking the mouse as the input device, the user presses and moves the mouse so that the mouse pointer slides over the left edge of the display screen of the PC, then the position of the device located around the PC can be recognized by using the positioning technology such as Bluetooth, Ultra-wideband (UWB), ultrasonic wave, etc., such as the PC recognizes that the mobile phone is located on the left side of the PC, then the mouse can be determined to shuttle to the mobile phone.

[0190] The shuttle relationship can be configured by the user in advance through a configuration file, or a configuration interface for configuring the shuttle relationship can be provided for the user, and the user can configure the shuttle relationship in advance through the configuration interface. For example, taking the configuration of the shuttle relationship of the mobile phone by the user through the interface as an example. The PC receives the operation of the user opening the configuration interface, and displays the configuration interface. The configuration interface includes the identifier (such as the icon) of the PC and the identifier (such as the icon) of the mobile phone. The user can configure the shuttle relationship by dragging the two identifiers. As an example, if the user places the identifier of the mobile phone to the left of the identifier of the PC, the PC can determine that when the mouse pointer slides over the left edge of the PC display screen, the mouse is shuttled to the mobile phone. If the user places the identifier of the mobile phone to the right of the identifier of the PC, the PC can determine that when the mouse pointer slides over the right edge of the PC display screen, the mouse is shuttled to the mobile phone. When there are multiple devices, the shuttle relationship of each device can be configured in advance. The following embodiments are described by taking the determined mouse shuttle to the mobile phone as an example. It should be noted that for the two implementations of determining the mouse to shuttle to which device according to the preconfigured shuttle relationship and the recognized device position, the above S301 can be executed before the mouse shuttle trigger or after the mouse shuttle trigger, which is not limited in this embodiment.

[0191] For example, in the process in which the user inputs a movement operation by using the input device of the PC, such as the movement of the mouse, the PC keyboard and mouse module can receive a corresponding input event, such as a movement event, which can be referred to as a mouse movement event. Since the mouse shuttle condition has not been triggered at this time, the input event, that is, the mouse movement event, will not be intercepted by the hook, and will be transmitted to the windows system of the PC. According to the received mouse movement event, the windows system of the PC can draw an animation of the movement of the mouse pointer and display it on the display screen of the PC. For example, as shown in FIG. 4A, with the movement of the mouse 401, the PC displays the animation of the movement of the mouse pointer 403 on the display screen 402 of the PC, as shown in FIG. 4B. Figure 5A Figure 5A As shown in FIG. 4B, the movement track of the mouse pointer 403 is shown as a track 404.

[0192] As described in S301, after the keyboard and mouse sharing mode is started, the PC starts input monitoring and mounts the hook. In the process in which the mouse pointer moves on the display screen of the PC, the PC keyboard and mouse module can monitor the real-time coordinate position of the mouse pointer on the display screen of the PC by using the input monitoring. When the PC keyboard and mouse module determines that the mouse pointer slides over the edge of the display screen of the PC according to the monitored real-time coordinate position of the mouse pointer on the display screen of the PC, it is determined that the mouse shuttle condition is met, indicating that the user wants to use the mouse to control other terminals. The PC keyboard and mouse module can determine that the mouse shuttle starts.

[0193] ​After the PC's keyboard and mouse module determines that the mouse shuttle has started, if the user operates the PC's input device, such as a mouse, the PC's keyboard and mouse module will intercept the input device, such as a mouse event, using a hook, and capture the operating parameters, such as mouse operating parameters, in the intercepted input event using the hook, and then transmit the operating parameters to the mobile phone through the PC's transmission management module through the established connection for the mobile phone to make a corresponding response. For example, in combination with the example shown in Figure 5A After the mouse pointer slides over the edge of the PC display screen, the user continues to move the mouse in the same direction. The PC's keyboard and mouse module can receive a movement event, such as a mouse movement event. Since the mouse shuttle has started, the PC's keyboard and mouse module can intercept (or shield) it using a hook, so that the mouse movement event will not be sent to the PC's Windows system, so that the PC will not respond to the received mouse movement event. The PC's keyboard and mouse module can also capture the operating parameters, such as mouse operating parameters (such as a mouse operating parameter 1, which can be a third operating parameter in this application), in the mouse movement event using a hook, and send the mouse operating parameter 1 to the mobile phone through the established connection through the PC's transmission management module. For example, as an example, the mouse operating parameter 1 corresponding to the mouse movement event can be a mouse key flag bit indicating that the user has moved the mouse, coordinate information indicating the X and Y coordinates of the mouse movement, scroll wheel information (value is empty), and key information (value is empty).

[0194] In addition, after the PC's keyboard and mouse module determines that the mouse shuttle has started, the PC's keyboard and mouse module can send shuttle state information (which can be first shuttle state information in this application) indicating that the mouse has started to shuttle to the mobile phone through the established connection through the PC's transmission management module. After the mobile phone receives this information, it can simulate a mouse pointer and display the mouse pointer on the mobile phone's display screen (where the mouse pointer displayed on the mobile phone's display screen can be a first cursor in this application). For example, after the mobile phone's transmission management module receives this information, it transmits the information to the mobile phone's drag management module and launcher, so that the mobile phone's drag management module can create a mouse pointer and display the mouse pointer through the mobile phone's launcher. The PC's keyboard and mouse module can also hide the mouse pointer displayed on the PC's display screen to give the user a visual effect of the mouse pointer being shuttled from the PC to the mobile phone.

[0195] S305, the mobile phone receives the mouse operating parameter 1, simulates a mouse movement event according to the mouse operating parameter 1, and displays an animation of the mouse pointer moving on the mobile phone's display screen according to the mouse movement event.

[0196] After the mobile phone receives the operation parameter, such as the mouse operation parameter 1, the mobile phone can simulate a movement event, such as a mouse movement event, according to the mouse operation parameter 1. In response to the mouse movement event, the mobile phone can draw an animation of the movement of the mouse pointer and display the animation on the display screen of the mobile phone.

[0197] Generally, for applications installed in the mobile phone, the mobile phone can display icons of the applications on the desktop (or referred to as the home screen) of the mobile phone. In addition, as described in S304, the user triggers the mouse pointer to switch from the display screen of the PC to the display screen of the mobile phone for the purpose of dragging the icon of the application displayed on the display screen of the mobile phone to the PC. Therefore, after the user moves the mouse to switch the mouse pointer from the display screen of the PC to the display screen of the mobile phone, the user continues to move the mouse, i.e., the user uses the mouse of the PC to input the operation of moving the mouse pointer on the mobile phone until the mouse pointer moves to the position of the icon of the application that the user wants to drag on the display screen of the mobile phone.

[0198] It should be noted that the application described in the embodiment can be an embedded application (i.e., a system application of the mobile phone, such as a calculator, a camera, a setting, a gallery, etc.), or a downloadable application (such as a browser, a weather, an email, etc.). The embedded application is an application program provided as part of the implementation of the mobile phone. The downloadable application is an application program that can provide its own internet protocol multimedia subsystem (IMS) connection. The downloadable application can be an application pre-installed in the mobile phone or a third-party application downloaded and installed in the mobile phone by the user.

[0199] For example, after the transmission management module of the mobile phone receives the operation parameter, such as the mouse operation parameter 1, the transmission management module of the mobile phone can transmit the mouse operation parameter 1 to the drag management module of the mobile phone. Since the PC and the operating system of the mobile phone are different, the key values of the mouse operation parameter in the input event, such as the mouse event, are different. Therefore, the drag management module of the mobile phone can convert the key code of the received mouse operation parameter 1 to a key code recognizable by the mobile phone according to a preset mapping relationship. Then, the mobile phone can simulate an input device recognizable by the mobile phone, such as a mouse event, according to the mouse operation parameter 1 after the conversion of the key code, i.e., can simulate a movement event, such as a mouse movement event. The drag management module of the mobile phone can draw an animation of the movement of the mouse pointer according to the simulated mouse movement event and display the animation on the display screen of the mobile phone by the launcher of the mobile phone. For example, as shown in FIG. 6, the mobile phone can display the animation of the movement of the mouse pointer on the display screen of the mobile phone. Figure 5BAs shown, taking the icon 504 of the calculator displayed on the display screen of the mobile phone as an example, which is the icon of the application that the user wants to drag. With the movement of the mouse 401, the mobile phone displays the animation corresponding to the movement of the mouse pointer 502 on the display screen 501 of the mobile phone, and the movement track of the mouse pointer 502 is shown as the track 503. That is, with the movement of the mouse 401, the mouse pointer 502 can move along the track 503 to the position of the icon 504 of the calculator.

[0200] After the mouse pointer moves to the position of the object, such as the icon of the application, that the user wants to drag, the user can input a drag operation through the mouse of the PC, so that the mobile phone can implement the dragging of the icon of the application according to the drag operation. The drag operation can be an operation for indicating the initiation of the dragging of the object, such as the icon of the application. The drag operation can include one operation or multiple operations. For example, the drag operation can include two operations, which are a pressing operation and a moving operation. Taking the mouse as an example, the pressing operation can be a mouse pressing operation, and the moving operation can be a mouse moving operation. The process that the user drags the icon of the application on the mobile phone using the mouse of the PC can include the following S306-S309.

[0201] S306, the PC receives a mouse pressing event, intercepts the mouse pressing event, and sends the mouse operation parameter 2 contained in the mouse pressing event to the mobile phone.

[0202] The mouse operation parameter 2 can be the first operation parameter in the present application.

[0203] S307, the mobile phone receives the mouse operation parameter 2, simulates a mouse pressing event according to the mouse operation parameter 2, and obtains the information of the application according to the mouse pressing event.

[0204] The pressing event, such as the mouse pressing event, is an event received by the PC after the user performs a pressing operation on the input device, such as the mouse, of the PC when the mouse pointer moves to the display position of the icon of the application. It should be noted that, taking the mouse as an example, the pressing operation on the mouse, or the mouse pressing operation, can be a pressing operation on the left mouse button, or a pressing operation on the right mouse button, which is not specifically limited in the present embodiment.

[0205] In combination with the description of S305, Figure 5B and S305, the user wants to drag the icon of the application displayed on the display screen of the mobile phone (such as the icon 504 of the calculator, which is the icon of the application that the user wants to drag), and the mouse pointer 502 moves to the display position of the icon 504 of the calculator. Figure 5BThe icon 504 of the calculator in the shown example is dragged from the mobile phone to the PC. Taking the mouse as an example of the input device, when the user moves the mouse pointer to the position of the icon 504 of the calculator by moving the mouse, the user can perform a pressing operation on the mouse (e.g., the left button of the mouse). The mouse module of the PC can receive the pressing event, such as a mouse pressing event. Since the mouse pressing event is received by the PC after the mouse shuttle starts, the mouse module of the PC uses the HOOK to intercept (or shield) the mouse pressing event so that the mouse pressing event is not sent to the windows system of the PC, thereby causing the PC to not respond to the received mouse moving event. The mouse module of the PC can also use the HOOK to capture the operation parameters of the mouse pressing event, such as mouse operation parameters, e.g., mouse operation parameters 2, and send the captured mouse operation parameters 2 to the mobile phone through the transmission management module of the PC through the established connection. The mouse operation parameters 2 can include a mouse button flag bit indicating that the user has pressed the mouse, coordinate information (with a value of null), scroll wheel information (with a value of null), and key information indicating that the user has operated the left button of the mouse.

[0206] The transmission management module of the mobile phone can receive the mouse operation parameters 2 and send the mouse operation parameters 2 to the drag management module of the mobile phone. After receiving the mouse operation parameters 2, the drag management module of the mobile phone can convert the key code of the received mouse operation parameters 2 into a key code recognizable by the mobile phone according to a preset mapping relationship, and then simulate a pressing event recognizable by the mobile phone, such as a mouse pressing event, according to the converted mouse operation parameters 2 using the created virtual input device. Subsequently, the drag management module of the mobile phone can determine that the user wants to drag the icon (e.g., the icon of the calculator) of the application at the current position of the mouse pointer according to the current display position of the mouse pointer. For example, after the mouse shuttle starts (e.g., the mobile phone receives the shuttle state information from the PC indicating that the mouse starts to shuttle), the mobile phone can register a listener for the coordinate position of the mouse pointer. Through the listener, the mobile phone can monitor the coordinate position of the mouse pointer on the display screen of the mobile phone in real time. That is, the drag management module of the mobile phone can determine the current coordinate position of the mouse pointer using the listener. Subsequently, the mobile phone can obtain the information of the application. For example, the launcher of the mobile phone can obtain the information of the application and send the obtained information of the application to the drag management module of the mobile phone. The drag management module of the mobile phone saves the received information of the application. The information of the application can include one or more of the following information: the name of the application, the icon of the application, the package name of the application, the class name of the application, the size of the icon of the application, and the display position of the icon of the application on the display screen of the mobile phone.

[0207] In addition, in the embodiment, after the phone simulates the press event, such as the mouse press event, and determines that the user performs the press operation, such as the mouse press operation, on the icon of the application, the drag management module of the phone can further create an icon (referred to as a drag icon, which can be a drag object in the application) same as the icon of the application for subsequent dragging. The drag management module of the phone can appropriately adjust the transparency of the drag icon, or can not adjust the transparency. Optionally, the drag management module of the phone can hide the icon (referred to as an original icon) of the application displayed on the display screen of the phone, or can not hide the icon, but continue to display the icon on the display screen of the phone by the launcher of the phone.

[0208] S308, the PC receives the mouse movement event, intercepts the mouse movement event, and sends the mouse operation parameter 3 contained in the mouse movement event to the phone.

[0209] The mouse operation parameter 3 can be a second operation parameter in the application.

[0210] S309, the phone receives the mouse operation parameter 3, simulates the mouse movement event according to the mouse operation parameter 3, and displays the animation of the icon of the application following the mouse pointer on the display screen of the phone as a response to the mouse press event and the mouse movement event.

[0211] When the user wants to drag the icon of the application, in the description of S307, the user can not lift the finger after the press operation on the mouse, and move the mouse while continuing the press operation on the mouse. At this time, the PC keyboard and mouse module can receive the movement event, such as the mouse movement event. Similarly, since the mouse movement event is received by the PC after the mouse shuttle starts, the PC keyboard and mouse module will use the hook to intercept (or shield) it, so that the mouse movement event will not be sent to the windows system of the PC, so that the PC will not respond to the received mouse movement event. The PC keyboard and mouse module can also capture the operation parameters, such as the mouse operation parameters, contained in the mouse movement event, such as the mouse operation parameter 3, and send it to the phone through the transmission management module of the PC through the established connection.

[0212] The transmission management module of the mobile phone can receive the mouse operation parameter 3 and send the mouse operation parameter 3 to the drag management module of the mobile phone. After receiving the mouse operation parameter 3, the drag management module of the mobile phone can simulate corresponding movement events, such as mouse movement events, after key code conversion of the mouse operation parameter 3. Then, in combination with the simulated press event and movement event (such as mouse press event and mouse movement event), the drag management module of the mobile phone can initiate the dragging of the icon of the application, and can also draw an animation of the icon of the application following the mouse pointer, and hand over to the launcher of the mobile phone for display on the display screen of the mobile phone.

[0213] For example, in combination with the example of Figure 5B , after the user presses the mouse at the position of the calculator icon with the mouse pointer, the user does not lift the finger and continues to move the mouse while pressing the mouse. Then, as shown in Figure 6 , as the mouse 401 moves, the mobile phone displays an animation of the calculator icon 602 following the mouse pointer 603 on the display screen 601 of the mobile phone, such as the trajectory of the calculator icon 602 following the mouse pointer 603 as shown by the trajectory 604. In some embodiments, the calculator icon 602 following the mouse pointer 603 can be the drag icon described in the description of S307, that is, the icon created (or copied) by the mobile phone which is the same as the calculator icon displayed on the desktop. It can also be understood that when the mobile phone does not hide the original icon of the calculator, after the drag icon is dragged, the original icon of the calculator and the drag icon following the mouse pointer 603 are displayed on the display screen of the mobile phone. It should be noted that in some embodiments, only the icon of the application can move as the mouse moves, that is, only the icon of the application is dragged. In other embodiments, as the mouse moves, the icon of the application and the name of the application are dragged together, that is, the icon of the application and the name of the application are dragged together (as shown in Figure 6 ). In embodiments where the icon of the application and the name of the application are dragged together, the mobile phone also needs to create a name identical to the name of the application being dragged when creating the drag icon.

[0214] In this embodiment, during the dragging of the drag icon, the mobile phone can not perform the process of finding a new placement position for the icon, nor perform the page turning of the desktop, but perform the following S310-S311 so that the drag icon can be dragged to the display screen of the PC.

[0215] S310, the mobile phone monitors the coordinate position of the mouse pointer on the display screen of the mobile phone.

[0216] S311, the mobile phone sends the shuttle state information indicating the end of the mouse shuttle to the PC and sends the icon of the application according to the coordinate position of the mouse pointer on the display screen of the mobile phone when determining that the mouse pointer slides out of the edge of the display screen of the mobile phone.

[0217] In the process of moving the icon of the application along with the mouse pointer on the display screen of the mobile phone, the mobile phone can determine whether the dragged icon of the application is dragged out of the edge of the display screen of the mobile phone. In the embodiment, the icon of the application being dragged out of the edge of the display screen of the mobile phone can mean that part of the icon of the application is dragged out (or overflowed) of the display screen of the mobile phone (i.e. the proportion of the icon of the application overflowing the display screen is greater than a preset threshold), or that the whole icon of the application is dragged out (or overflowed) of the display screen of the mobile phone, or that the mouse pointer slides out of the edge of the display screen of the mobile phone when the icon of the application moves along with the mouse pointer on the display screen of the mobile phone. The embodiment does not make specific limitation here. The following is described by taking the icon of the application being dragged out of the edge of the display screen of the mobile phone as an example.

[0218] In the process of moving the icon of the application along with the mouse pointer on the display screen of the mobile phone, the mobile phone (such as the transmission management module of the mobile phone) can monitor the real-time coordinate position of the mouse pointer on the display screen of the mobile phone (as described in S307, the real-time coordinate position of the mouse pointer can be acquired by using the registered listener). For example, the mobile phone can determine the coordinate position of the mouse pointer on the display screen of the mobile phone according to the initial position and the relative displacement of the mouse pointer, so as to determine whether the mouse pointer slides out of the edge of the display screen of the mobile phone. The initial position of the mouse pointer can be the coordinate position of the mouse pointer on the display screen of the mobile phone when the mouse starts to move, or the coordinate position of the mouse pointer on the display screen of the mobile phone before the mouse starts to move. The initial position of the mouse pointer can specifically mean the coordinate position in the coordinate system taking the top-left corner of the display screen of the mobile phone as the coordinate origin, the X axis pointing to the right edge of the display screen of the mobile phone from the top-left corner, and the Y axis pointing to the lower edge of the display screen of the mobile phone from the top-left corner. The specific implementation of the mobile phone determining that the mouse pointer slides out of the edge of the display screen of the mobile phone is similar to the specific implementation of the PC determining that the mouse pointer slides out of the edge of the display screen of the PC, which will not be described in detail here.

[0219] When the phone's transmission management module determines, based on the real-time coordinates of the mouse pointer on the phone's screen, that the mouse pointer has moved across the edge of the screen, it indicates that the user wants to control other devices with the mouse. As described in S304, if the phone is connected to only one device, the PC, it indicates that the user wants to control the PC with the mouse. If the phone is connected to multiple devices, it can display a list of options, including the identifiers of all devices connected to the phone, allowing the user to select the device they want to control with the mouse. If the user selects the PC identifier, it indicates that the user wants to control the PC with the mouse. Alternatively, a routing relationship can be pre-configured in the phone to determine which device the mouse will route to, i.e., which device will respond to the mouse operation. The specific descriptions of the routing relationship configuration and application are similar to those in the above embodiments and will not be elaborated upon here.

[0220] When the phone's transmission management module determines that the user intends to control the PC using a mouse, it can determine that mouse hovering has ended. At this time, the phone's transmission management module can send hovering status information (which can be the second hovering status information described in this application) to the PC through the established connection, indicating the end of mouse hovering. The phone's transmission management module can also indicate the end of mouse hovering to the phone's drag management module, so that the drag management module can send drag data to the PC through the established connection via the phone's transmission management module. This drag data may include an application icon. Before sending the application icon to the PC, the phone can also serialize the application icon; that is, the application icon sent to the PC by the phone can be data obtained after serialization. If the application name and application icon are dragged together, the phone also needs to send the application name to the PC.

[0221] Additionally, once the mouse pointer moves across the edge of the phone screen, indicating the mouse hover is complete, the dragging of the application icon on the screen ends. The phone's drag management module can also indicate the end of the mouse hover to the phone's launcher, so that the launcher can display a drag-end animation on the phone's screen, such as showing the dragged icon moving along a trajectory from the edge of the screen (e.g.,...). Figure 6 The animation shown (track 604, direction opposite to track 604) returns to the original position. The phone does not rearrange the app icons displayed on the screen.

[0222] It should be noted that the above embodiment is described by taking the case that the mobile phone sends the drag data to the PC after determining the end of the mouse shuttle as an example. In some other embodiments, the mobile phone can temporarily cache the drag data, such as the information of the object (e.g., the icon of the application) at the beginning of the drag. After determining the end of the mouse shuttle, the mobile phone sends the shuttle state information indicating the end of the mouse shuttle to the PC. After receiving the shuttle state information, the PC sends a message requesting the drag data to the mobile phone. After receiving the message, the mobile phone sends the drag data to the PC.

[0223] S312, the PC receives the mouse movement event, and displays an animation of the icon of the application following the movement of the mouse pointer on the display screen of the PC according to the mouse movement event and the received icon of the application.

[0224] After the transmission management module of the PC receives the shuttle state information indicating the end of the mouse shuttle from the mobile phone, the transmission management module of the PC indicates the end of the mouse shuttle to the drag management module of the PC and the keyboard and mouse module of the PC. The PC can determine that the user needs to return to the response of the mouse operation by the PC. Then, the PC unloads the HOOK, i.e., cancels the interception of the input event, such as the mouse event. After that, when the user operates the input device of the PC, the keyboard and mouse module of the PC will not intercept the received input event, but send the received input event to the windows system of the PC, so that the windows system of the PC responds to the input event. The keyboard and mouse module of the PC can also display the mouse pointer on the display screen of the PC again.

[0225] For example, taking the input device as a mouse as an example. The user continues to press the mouse while moving the mouse. The PC can receive a movement event, such as a mouse movement event. Since the HOOK has been unloaded at this time, the keyboard and mouse module of the PC will send the received mouse movement event to the windows system of the PC, so that the windows system of the PC responds accordingly. For example, the PC can draw the icon of the application according to the mouse movement event and the received icon of the application, display the icon of the application in the form of a window, draw an animation of the icon of the application following the movement of the mouse pointer, and display the animation of the icon of the application following the movement of the mouse pointer on the display screen of the PC. If the name of the application and the icon of the application are dragged together, the PC also needs to draw the name of the application, display the name of the application in the form of a window, and move following the movement of the mouse pointer. For example, in combination with Figure 6 As shown in Figure 7 , as the mouse 401 moves, the PC displays an animation of the icon 702 of the calculator following the movement of the mouse pointer 703 on the display screen 701 of the PC, such as the trajectory of the icon 702 of the calculator following the movement of the mouse pointer 703 shown as the trajectory 704. That is, in combination with Figure 6 and Figure 7As shown, the user can visually achieve the effect of dragging the icon of the application (e.g., the icon of the calculator) displayed on the display screen of the mobile phone from the display screen of the mobile phone to the display screen of the PC by long-pressing the mouse and moving the mouse. Of course, if the icon of the application is subjected to serialization processing before being sent to the PC, the PC can obtain the icon of the application by performing deserialization processing after receiving the corresponding data.

[0226] In addition, in S311, the mobile phone can also send the size of the icon of the application obtained in S307 and the display position of the icon of the application on the display screen of the mobile phone to the PC. According to these data, the PC can determine the size and initial position of the window for displaying the icon of the application by using a preset rule. As an example, the preset rule can be that the size of the window is the same as the size of the icon of the application obtained in S307, and the initial position of the window is the same as the display position of the icon of the application on the display screen of the mobile phone obtained in S307 (for example, the display position of the icon of the application on the display screen of the mobile phone obtained in S307 is the lower right corner of the display screen of the mobile phone, and the initial position of the window is the lower right corner of the display screen of the PC). Alternatively, the size of the window is twice the size of the icon of the application obtained in S307, the initial position of the window is the same as the display position of the icon of the application on the display screen of the mobile phone obtained in S307, and the like. The embodiments of the present application do not make specific limitations on the rule here.

[0227] S312 is an optional step.

[0228] S313, the PC receives the mouse-up event and displays the interface of the application on the display screen of the PC according to the mouse-up event.

[0229] When the user drags the icon of the application (e.g., the icon of the calculator) to be dragged to the display screen of the PC, the user can input a drag release operation to end the dragging of the icon of the application. For example, the user can lift the finger (i.e., the drag release operation can be the operation of lifting the finger, or in other words, stopping the long-press operation on the mouse). The user can also stop moving the mouse. The mouse module of the PC can receive the lifting event, such as the mouse-up event. Similarly, since the hook has been unloaded at this time, the mouse module of the PC will send the received lifting event, such as the mouse-up event, to the windows system of the PC for response. For example, in response to the mouse-up event, the PC can display the interface of the dragged application at the position where the mouse pointer stops moving on the display screen of the PC. For example, in combination with the example of Figure 7 Figure 8 ​As shown, when the user drags the calculator icon to the display screen of the PC, the user lifts the finger and stops moving the mouse. At this time, in response, the PC can display the interface 802 of the calculator on the display screen 801 of the PC. In addition, for the mobile phone, after the user lifts the finger and stops moving the mouse, the interface displayed by the mobile phone can not be changed, as shown in (a) of FIG. 8, the desktop 803 is still displayed, and the mobile phone can also display the interface of the calculator, as shown in (b) of FIG. 8, the mobile phone displays the interface 804 of the calculator. Figure 8 Figure 8

[0230] As an example, the specific implementation that the PC displays the interface of the application on the display screen of the PC according to the mouse-up event can be: after the mouse module of the PC receives the lift event, such as the mouse-up event, the mouse module of the PC can send a message to the mobile phone through the transmission management module of the PC through the connection established with the mobile phone. The message can be a request message for requesting the interface of the application, or the message can be an indication message for indicating that the dragging of the icon of the application is ended. After the mobile phone receives the message, the mobile phone can open the application according to the package name and the class name of the application in the application information obtained in S307, and send the interface of the opened application to the PC through the connection established with the PC, so that the PC can display the interface of the application on the display screen of the PC. For example, the PC and the mobile phone can use wireless projection (Miracast) to realize the display of the interface of the application on the display screen of the PC. That is, after the mobile phone receives the message, the mobile phone can integrate all layers of the obtained interface of the application into a video stream (or called screen recording data) and encode the video stream, and then send the video stream to the PC through the real time streaming protocol (RTSP) protocol. After the PC receives the video stream, the PC can decode and play the video stream, so as to realize the display of the interface of the application on the display screen of the PC. For another example, the mobile phone can use distributed multi-media protocol (DMP) technology to realize the display of the interface of the application on the display screen of the PC. For example, after the mobile phone receives the message, the mobile phone can create a virtual display (VirtualDisplay). Then, the mobile phone can move the drawing of the obtained interface of the application to the VirtualDisplay. The mobile phone can perform screen recording processing on the interface of the application drawn on the VirtualDisplay. In this way, the mobile phone can obtain the screen recording data of the interface of the application. The mobile phone can encode the obtained screen recording data and send the screen recording data to the PC. The PC can receive the corresponding data, decode the data, and then obtain the screen recording data. The PC can realize the display of the interface of the application on the display screen of the PC according to the screen recording data obtained after decoding.

[0231] ​​The method provided in the embodiment can increase the interaction between terminals in the multi-terminal cooperative use, and improve the user experience.

[0232] Figure 9 Another flowchart of the cross-device object dragging method provided in the embodiment is shown in the figure. In the embodiment, the implementation process of dragging the interface of an application displayed on a mobile phone from the mobile phone to a PC by using the mouse shared by the PC is introduced. As shown in the figure, Figure 9 The method can include the following S901-S913.

[0233] S901, the mobile phone establishes a connection with the PC.

[0234] S902, the PC receives a mouse movement event, and displays an animation of the movement of a mouse pointer on the display screen of the PC according to the mouse movement event.

[0235] S903, the PC monitors the coordinate position of the mouse pointer on the display screen of the PC.

[0236] S904, the PC intercepts the mouse movement event according to the coordinate position of the mouse pointer on the display screen of the PC, and sends the mouse operation parameter 1 contained in the mouse movement event to the mobile phone when it is determined that the mouse pointer slides out of the edge of the display screen of the PC.

[0237] It should be noted that the description of other related contents in S901-S904 can refer to the description of the corresponding contents in S301-S304 in the embodiment shown in the figure, which will not be described here. Figure 3

[0238] S905, the mobile phone receives the mouse operation parameter 1, simulates a mouse movement event according to the mouse operation parameter 1, and displays an animation of the movement of a mouse pointer on the display screen of the mobile phone according to the mouse movement event.

[0239] The mouse operation parameter 1 is the third operation parameter in the application.

[0240] After the mobile phone receives the mouse operation parameter 1, a movement event such as a mouse movement event can be simulated according to the mouse operation parameter 1. In response to the mouse movement event, the mobile phone can draw an animation of the movement of a mouse pointer and display it on the display screen of the mobile phone. That is, the user can trigger the mouse pointer to shuttle from the display screen of the PC to the display screen of the mobile phone by moving the mouse.

[0241] ​The user triggers the mouse pointer to shuttle from the PC display screen to the mobile phone display screen, and the purpose is to want to drag the interface of the current application displayed on the mobile phone display screen from the mobile phone to the PC. In this embodiment, the operation of triggering the application interface windowing can be a predefined operation, such as a long-press operation on the status bar. Therefore, after the user moves the mouse to shuttle the mouse pointer from the display screen of the PC to the display screen of the mobile phone, the user will continue to move the mouse until the mouse pointer moves to the position of the status bar displayed on the display screen of the mobile phone.

[0242] For example, the user wants to drag the interface of the calculator displayed on the mobile phone to the display screen of the PC. As shown in Figure 10 The display screen of the mobile phone currently displays the interface 1001 of the calculator. After the user moves the mouse 1002 to trigger the mouse pointer to shuttle from the display screen of the PC to the display screen of the mobile phone, the user will continue to move the mouse 1002. With the movement of the mouse 1002, the mobile phone displays an animation corresponding to the movement of the mouse pointer 1003 on the display screen of the mobile phone, such as the movement track of the mouse pointer 1003 shown as track 1004. That is, with the movement of the mouse 1002, the mouse pointer 1003 can move along the track 1004 to the position of the status bar 1005 displayed on the mobile phone.

[0243] S906, the PC receives the mouse down event and the mouse up event, intercepts the mouse down event and the mouse up event, and sends the mouse operation parameter 2 of the mouse down event and the mouse operation parameter 3 of the mouse up event to the mobile phone.

[0244] The mouse operation parameter 2 is the fourth operation parameter in the present application, and the mouse operation parameter 3 is the fifth operation parameter in the present application.

[0245] S907, the mobile phone receives the mouse operation parameter 2 and the mouse operation parameter 3, simulates the mouse down event according to the mouse operation parameter 2, simulates the mouse up event according to the mouse operation parameter 3, displays a window on the display screen of the mobile phone according to the mouse down event and the mouse up event, the window includes the interface screenshot of the application, and obtains the window information.

[0246] In combination with Figure 10 and the description of S905, the user wants to drag the interface of the application displayed on the display screen of the mobile phone (such as Figure 10The interface 1001 of the calculator in the illustrated example is dragged from the mobile phone to the PC, and the operation of triggering the application interface windowing is taken as an example of a long-press operation on the status bar. When the user moves the mouse pointer to the position of the status bar by moving the mouse, the user can perform a press operation on the mouse (such as the left mouse button) and then lift the finger after a certain time. The PC key mouse module can receive the press event (such as the mouse press event) and the lift event (such as the mouse lift event) and intercept them. The PC key mouse module can also capture the operation parameters (such as mouse operation parameters 2 and mouse operation parameters 3) of the intercepted mouse press event and mouse lift event through the transmission management module of the PC and send them to the mobile phone through the established connection.

[0247] The transmission management module of the mobile phone can receive the mouse operation parameters 2 and the mouse operation parameters 3 and send them to the drag management module of the mobile phone. After receiving the mouse operation parameters 2 and the mouse operation parameters 3, the drag management module of the mobile phone converts the key code of the received mouse operation parameters 2 into a key code recognizable by the mobile phone according to the preset mapping relationship, converts the key code of the received mouse operation parameters 3 into a key code recognizable by the mobile phone, and then simulates a press event such as a mouse press event using the created virtual input device according to the converted mouse operation parameters 2 and simulates a lift event such as a mouse lift event using the created virtual input device according to the converted mouse operation parameters 3. After that, the drag management module of the mobile phone can determine that the user wants to drag the interface of the currently displayed application (such as the interface of the calculator) according to the current display position of the mouse pointer. In response to the mouse press event and the mouse lift event, the systemUI of the mobile phone sends a status bar event to the launcher of the mobile phone. After receiving the status bar event, the launcher of the mobile phone can window the interface of the currently displayed application, that is, display a window on the display screen of the mobile phone, and the window includes the interface of the application, such as a screenshot of the application interface (such as a screenshot of the calculator interface). The size of the window can be a pre-set value. In addition, the mobile phone (such as the launcher of the mobile phone) can also perform background blurring processing. In this embodiment, the background can be the interface of the application or the desktop of the mobile phone, which is not limited in this embodiment. For example, in combination with Figure 10 Figure 11 As shown, after simulating the mouse press event and the mouse lift event, the mobile phone can display a window 1101 on the display screen of the mobile phone, and the window 1101 includes a screenshot of the interface 1001 of the calculator. The mobile phone also performs background blurring processing, Figure 11 In this embodiment, the background is taken as an example of the interface 1001 of the computer.

[0248] ​The mobile phone (such as the launcher of the mobile phone) can also obtain window information (the window information is the object information in the present application). The window information can include one or more of the following information: the name of the application in the window, the package name of the application, the class name of the application, the interface screenshot of the application, the size of the window, and the display position of the window on the display screen of the mobile phone.

[0249] In S908, the PC receives the mouse moving event and the mouse pressing event, intercepts the mouse moving event and the mouse pressing event, and sends the mouse operation parameter 4 of the mouse moving event and the mouse operation parameter 5 of the mouse pressing event to the mobile phone.

[0250] In S909, the mobile phone receives the mouse operation parameter 4 and the mouse operation parameter 5, simulates the mouse moving event according to the mouse operation parameter 4, simulates the mouse pressing event according to the mouse operation parameter 5, and displays the animation of the window following the mouse pointer on the display screen of the mobile phone according to the mouse moving event and the mouse pressing event.

[0251] The mouse operation parameter 4 can be the first operation parameter in the present application. The mouse operation parameter 5 can be the second operation parameter in the present application.

[0252] When the user wants to drag the interface of the application, the user can move the mouse to trigger the mouse pointer to move from the status bar of the display screen of the mobile phone to the window position in S907. Then, the user can perform a long-press operation on the mouse while moving the mouse. At this time, the PC keyboard and mouse module can receive the pressing event (such as the mouse pressing event) and the moving event (such as the mouse moving event). Similarly, the PC keyboard and mouse module will intercept it and capture the corresponding operation parameters (such as the mouse operation parameter 4 and the mouse operation parameter 5) and then send them to the mobile phone through the transmission management module of the PC through the established connection.

[0253] The transmission management module of the mobile phone can receive the mouse operation parameter 4 and the mouse operation parameter 5 and send them to the drag management module of the mobile phone. After receiving the mouse operation parameter 4 and the mouse operation parameter 5, the drag management module of the mobile phone can convert the key code of the received mouse operation parameter 4 into a key code that can be recognized by the mobile phone according to the preset mapping relationship, and convert the key code of the received mouse operation parameter 5 into a key code that can be recognized by the mobile phone. Then, the virtual input device created by the mobile phone can simulate a pressing event (such as a mouse pressing event) according to the converted mouse operation parameter 4, and simulate a moving event (such as a mouse moving event) according to the converted mouse operation parameter 5. In response, the drag management module of the mobile phone can initiate a drag operation on the window, and can also draw an animation of the window following the mouse pointer and display it on the display screen of the mobile phone through the launcher of the mobile phone. For example, in combination with Figure 11Users can move the mouse to trigger the mouse pointer to move from the status bar on the phone screen. Figure 11 The position of the middle window. Then, as... Figure 12 As shown, the user can long-press the mouse 1201 and move the mouse 1201 simultaneously. As the mouse 1201 moves, the phone displays an animation on the phone screen of window 1202 (which includes a screenshot of the calculator interface) following the movement of the mouse pointer 1203, as shown in trajectory 1204.

[0254] As the window moves with the mouse pointer, the phone can determine whether the dragged window has been dragged off the edge of the phone screen. In this embodiment, the window being dragged off the edge of the phone screen can mean that a portion of the window is dragged off (or overflows) the phone screen (i.e., the proportion of the window overflowing the screen is greater than a preset threshold), or the entire area of ​​the window is dragged off (or overflows) the phone screen, or the mouse pointer slides off the edge of the phone screen as the window moves with the mouse pointer. This embodiment does not impose specific limitations here. The following explanation uses the determination of whether the dragged window has been dragged off the edge of the phone screen specifically as determining whether the mouse pointer slides off the edge of the phone screen as an example.

[0255] S910: Mobile phone monitors the coordinates of the mouse pointer on the mobile phone screen.

[0256] S911: Based on the coordinates of the mouse pointer on the phone screen, when the mouse pointer moves out of the edge of the phone screen, the phone sends a transit status information to the PC to indicate the end of the mouse transit, and sends a screenshot of the application interface to the PC.

[0257] Before sending the application's screenshot to the PC, the mobile phone can also perform serialization processing on it. In other words, the application's screenshot sent from the mobile phone to the PC can be data obtained after serialization processing. The shuttle state information in S911 can be the second shuttle state information in this application.

[0258] S912, the PC receives a mouse movement event, and based on the mouse movement event and the received screenshot of the application interface, displays an animation on the PC's screen showing the screenshot of the application interface following the movement of the mouse pointer.

[0259] For example, the user continues to press the mouse and moves the mouse. The PC can receive a movement event, such as a mouse movement event. The PC can draw the interface screenshot of the application according to the mouse movement event and the received interface screenshot of the application, and display the interface screenshot of the application in the form of a window. The PC can also draw an animation of the interface screenshot of the application moving with the mouse pointer, and display the animation of the interface screenshot of the application moving with the mouse pointer on the display screen of the PC. For example, in combination with Figure 12 As shown in Figure 13 , as the mouse 1301 moves, the PC displays an animation of the interface screenshot 1302 of the application moving with the mouse pointer 1303 on the display screen of the PC, such as the track of the interface screenshot 1302 of the application moving with the mouse pointer 1303 shown as the track 1304. In addition, after the mouse pointer slides over the edge of the display screen of the mobile phone, the dragging of the interface of the application on the display screen of the mobile phone ends, and the mobile phone can display an animation of the end of the dragging on the display screen of the mobile phone. For example, the mobile phone displays an animation of returning the interface of the application (or displays an animation of returning the desktop), such as Figure 13 As shown in Figure 13 , after the dragging on the mobile phone ends, the mobile phone displays the interface 1305 of the calculator.

[0260] That is, as shown in Figure 12 and Figure 13 , the user can visually achieve the effect of dragging the interface of the application (such as the interface of the calculator) displayed on the display screen of the mobile phone to the display screen of the PC by long-pressing the mouse and moving the mouse. Of course, if the interface screenshot of the application is serialized before being sent to the PC, the PC can obtain the interface screenshot of the application by performing deserialization processing after receiving the corresponding data.

[0261] In addition, in S911, the mobile phone can also send the size of the window obtained in S907 and the display position of the window on the display screen of the mobile phone to the PC. According to these data, the PC can determine the size and initial position of the window used to display the interface screenshot of the application by using a preset rule. As an example, the preset rule can be that the size of the window is the same as the size of the window obtained in S907, and the initial position of the window is the same as the display position of the window on the display screen of the mobile phone obtained in S907. Alternatively, the size of the window is twice the size of the window obtained in S907, and the initial position of the window is the same as the display position of the window on the display screen of the mobile phone obtained in S907, and the like. The present application does not specifically limit the rule here.

[0262] S913, the PC receives a mouse-up event, and displays the interface of the application on the display screen of the PC according to the mouse-up event.

[0263] When the user drags the interface of the application (e.g., the interface of the calculator) to be dragged to the display screen of the PC, the user can lift the finger (i.e., stop the long-press operation on the mouse) and stop moving the mouse. The PC can receive a lift event, such as a mouse lift event. In response to the mouse lift event, the PC can display the interface of the application at the position on the display screen of the PC where the mouse pointer stops moving. For example, in combination with the example of Figure 13 , as shown in Figure 14 , when the user drags the screenshot of the interface of the calculator to the display screen of the PC, the user lifts the finger and stops moving the mouse. At this time, in response, the PC can display the interface 1401 of the calculator on the display screen of the PC.

[0264] As an example, the specific implementation that the PC displays the interface of the application on the display screen of the PC according to the mouse lift event can be that, after the mouse lift event is received by the keyboard and mouse module of the PC, the keyboard and mouse module of the PC can send a request message to the mobile phone through the transmission management module of the PC through the connection established with the mobile phone, where the request message is used to request the interface of the application. After the request message is received by the mobile phone, the mobile phone can obtain the interface of the application according to the application name, the package name, and the class name in the window information obtained in S907, and send the interface to the PC through the connection established with the PC, so that the PC can display the interface of the application on the display screen of the PC. For example, the PC and the mobile phone can use Miracast to realize the display of the interface of the application on the display screen of the PC. For another example, the mobile phone can use DMP technology to realize the display of the interface of the application on the display screen of the PC.

[0265] It should be noted that the descriptions of other contents in the steps of this embodiment are similar to the descriptions of the corresponding contents in the embodiment shown in Figure 3 , and will not be repeated here.

[0266] The method provided in this embodiment increases the interaction between terminals when multiple terminals are used cooperatively, and improves the user experience when the user uses the same input device (e.g., a mouse) to control all the terminals.

[0267] It can be understood that, at present, when multiple terminals such as a PC and a mobile phone are used to realize collaborative office, the premise of realizing content dragging is to project the interface of the mobile phone to the display screen of the PC for display, and the dragging and processing of the content are all completed by using the hardware of the PC. The mobile phone is usually turned off, and its hardware capabilities (such as a display screen and a stylus) cannot be involved in the collaborative office. This will cause the user to have difficulty in comprehensively utilizing the characteristics of different terminals participating in the collaborative office, making the best of the strengths and avoiding the weaknesses, and maximizing the work efficiency. Moreover, the interface of the mobile phone projected to the PC will also occupy the display space of the display screen of the PC, affecting the use efficiency of the collaborative use of multiple terminals. The cross-device object dragging method and device provided in the embodiment can make the user use input devices such as a touchpad and a mouse to transfer content (or objects) such as text or files between multiple terminals participating in collaborative use by means of dragging, and allow the user to use these terminals to process the transferred content, without starting projection.

[0268] In other embodiments, the following is combined Figure 1 and Figure 2B The cross-device object dragging method provided in the embodiment of the application is described in detail by taking the first terminal 101 as a PC, the second terminal 102 as a tablet computer, and the input device 102-1 as a mouse as an example.

[0269] Figure 15 A flowchart of the cross-device object dragging method provided in the embodiment of the application is shown. In the embodiment, the method is described in detail by taking an example in which a user uses a mouse of a PC to transfer content in the PC to a tablet computer by means of dragging. Figure 15 As shown in the figure, the method can include the following S1501-S1510.

[0270] S1501, the tablet computer and the PC establish a connection.

[0271] After the tablet computer and the PC successfully establish a connection, the two can be used collaboratively. In order to improve the efficiency of collaborative use, the user can use a set of input devices, such as a mouse of the PC, to realize control of both the PC and the tablet computer.

[0272] In addition, since the mouse is an input device of the PC, after the other terminal is connected with the PC, the user can temporarily realize control of the PC by using the mouse. In the embodiment, after the PC starts the keyboard and mouse sharing mode, the PC can also trigger the other terminal connected with the PC to respond to the operation of the mouse when it is determined that the mouse shuttle condition is met. Exemplarily, the mouse shuttle condition can be that the mouse pointer corresponding to the mouse on the display screen of the PC slides over the edge of the display screen of the PC.

[0273] It should be noted that the specific description of the embodiment S1501 can refer to the specific description of the corresponding content in another embodiment S301 of the present application, and will not be described in detail here.

[0274] For example, if the user wants to transfer the content of the PC to the tablet computer by dragging, and continues to drag in the tablet computer, the process of triggering the other terminal connected with the PC, such as the tablet computer, to respond to the operation of the mouse created by the virtual input device can include the following S1502-S1506.

[0275] S1502, the PC receives the mouse operation of selecting the content.

[0276] The above-mentioned content can be text (or text), can also be a file, and can also be a folder. The file can include one or more of the following formats, such as a word document, an Excel workbook, a PowerPoint presentation, a bitmap, an image file, a text file, a sound file, a video file, a flash animation file, a web page file, a compressed file, etc.

[0277] In addition, in the present embodiment, the selected content can be one or more. For example, two word documents are selected. For another example, one word document and one image file are selected. For another example, two folders are selected.

[0278] S1503, the PC receives the mouse down event and the mouse move event, initiates the drag event according to the mouse down event and the mouse move event, and displays the animation of the shadow of the content moving with the mouse pointer on the display screen of the PC.

[0279] The mouse pointer of the PC can be the second cursor in the present application.

[0280] S1504, the PC monitors the coordinate position of the mouse pointer on the display screen of the PC.

[0281] S1505, the PC determines that the mouse pointer slides out of the edge of the display screen of the PC according to the coordinate position of the mouse pointer on the display screen of the PC, intercepts the mouse move event, and sends the mouse operation parameters contained in the mouse move event to the tablet computer.

[0282] S1506, the PC obtains the bitmap of the drag event content and the shadow, and sends the bitmap of the drag event content and the shadow to the tablet computer.

[0283] The drag event content is used for dragging the sink device, such as a tablet computer. For example, when the content in S1502 is text, the drag event content can include the text. For another example, when the content in S1502 is a file or a folder, the drag event content can include a file path (e.g., a uniform resource identifier (URI)). The drag data in this application can include the drag event content and a bitmap of the shadow, and can be used for the sink device, such as a tablet computer, to display an animation of the object moving with the mouse pointer on the display screen of the sink device.

[0284] In this embodiment, after the keyboard and mouse sharing mode is enabled, when a user wants to transfer the content of the PC to another terminal, such as a tablet computer, connected to the PC by creating a virtual input device, and the user continues to drag in the tablet computer, the user can use the input device of the PC to select the content to be dragged. Then, when the mouse pointer of the PC is displayed on the content, the user can input a drag operation, so that the PC can implement the drag operation on the corresponding object, i.e., the content (e.g., the selected content in S1502).

[0285] The drag operation can be an operation for indicating to initiate a drag event for the selected content. The drag operation can include one operation or multiple operations. For example, the drag operation includes two operations, i.e., a press operation and a move operation. Taking the input device as a mouse as an example, the press operation can be a mouse press operation, and the move operation can be a mouse move operation. The user can press and move the mouse (i.e., use the mouse of the PC to input the mouse press operation and the mouse move operation) to trigger the PC to initiate the drag event for the content, so that the content (e.g., a shadow of the content) can move with the mouse pointer of the PC on the display screen of the PC. The shadow can also be referred to as a drag shadow.

[0286] In the process that the content (e.g. the shadow of the content) follows the movement of the mouse pointer, the PC can determine whether the dragged content (e.g. the shadow of the content) is dragged out of the edge of the PC display screen. When the content (e.g. the shadow of the content) is dragged out of the edge of the PC display screen, the mouse shuttle condition can be triggered. In the embodiment, the content being dragged out of the edge of the PC display screen can be that part of the content (e.g. the shadow of the content) is dragged out of (or overflowed from) the PC display screen (i.e. the proportion of the window overflowed from the display screen area is greater than a preset threshold), or that the whole of the content (e.g. the shadow of the content) is dragged out of (or overflowed from) the PC display screen, or that the mouse pointer slides out of the edge of the PC display screen when the content (e.g. the shadow of the content) moves with the mouse pointer on the PC display screen, which is not specifically limited in the embodiment. The following is described by taking the example that whether the mouse pointer slides out of the edge of the PC display screen is determined as the example of whether the dragged content (e.g. the shadow of the content) is dragged out of the edge of the PC display screen.

[0287] As an example, the user can trigger the mouse shuttle condition by continuously moving the mouse in the same direction to make the mouse pointer corresponding to the display on the PC display screen slide over (or out of) the edge of the PC display screen.

[0288] As an example, the PC can determine the coordinate position of the mouse pointer on the PC display screen according to the initial position and the relative displacement of the mouse pointer, so as to determine whether the mouse pointer slides out of the edge of the PC display screen.

[0289] The initial position of the mouse pointer can be the coordinate position of the mouse pointer on the PC display screen when the mouse starts to move, or the coordinate position of the mouse pointer on the PC display screen before the mouse starts to move. The initial position of the mouse pointer can be specifically the coordinate position in the coordinate system with the top-left corner of the PC display screen as the coordinate origin, the X axis pointing to the right edge of the PC display screen from the top-left corner, and the Y axis pointing to the lower edge of the PC display screen from the top-left corner. For example, the specific process that the PC determines whether the mouse pointer slides out of the edge of the PC display screen can be: in combination with Figure 4 , the PC can establish the coordinate system with the initial coordinate position as the coordinate origin (e.g. the position o shown in Figure 4 ), the X axis pointing to the right edge of the PC display screen from the coordinate origin o, and the Y axis pointing to the upper edge of the PC display screen from the coordinate origin o. The PC can determine the coordinate values of the edges of the PC display screen in the coordinate system. The coordinate values of the edges of the PC display screen in the coordinate system can be determined according to the resolution of the PC display screen and the initial position of the mouse pointer. For example, as shown in Figure 4As shown, in the coordinate system, the coordinate value of the right edge of the PC display screen on the X axis is x1, the coordinate value of the left edge on the X axis is -x2, the coordinate value of the upper edge on the Y axis is y1, and the coordinate value of the lower edge on the Y axis is -y2. After the mouse moves, the mouse reports the relative displacement of the mouse pointer to the PC. The PC can calculate the coordinate position (x, y) of the mouse pointer on the PC display screen according to the relative displacement reported by the mouse. According to the coordinate position (x, y), the PC can determine whether the mouse pointer slides out of the edge of the PC display screen. For example, if the coordinate value x of the mouse pointer on the X axis is greater than x1, it can be determined that the mouse pointer slides out of the right edge of the PC display screen. If the coordinate value x of the mouse pointer on the X axis is less than -x2, it can be determined that the mouse pointer slides out of the left edge of the PC display screen. If the coordinate value y of the mouse pointer on the Y axis is greater than y1, it can be determined that the mouse pointer slides out of the upper edge of the PC display screen. If the coordinate value y of the mouse pointer on the Y axis is less than -y2, it can be determined that the mouse pointer slides out of the lower edge of the PC display screen.

[0290] After it is determined that the mouse pointer slides out of the edge of the PC display screen, that is, the mouse shuttle condition is triggered, the user can use the input device of the PC to control the other terminal connected with the PC, which creates a virtual input device. That is, after the mouse shuttle condition is triggered, the PC can send the data of the operation input by the user using the input device of the PC to the other terminal connected with the PC, which creates a virtual input device. For example, if the user continues to move the mouse in the same direction, the PC can intercept the received movement event, such as a mouse movement event, and transmit the operation parameter contained in the mouse movement event, such as a mouse operation parameter, to the other terminal connected with the PC, which creates a virtual input device, so that the terminal implements the continuation of the drag event.

[0291] It should be noted that in this embodiment, if the device connected with the PC is only one, such as the tablet computer described above, and the tablet computer creates a virtual input device, the PC can transmit the corresponding operation parameter to the tablet computer, so that the tablet computer implements the continuation of the drag event.

[0292] If there are multiple devices connected to the PC, and some or all of the multiple devices have created virtual input devices, in some embodiments, the PC can display a list option on the display screen of the PC when it determines that the mouse shuttle condition is triggered, the list option including the identification of the devices connected to the PC that have created virtual input devices (e.g., the identification of the tablet). The PC can determine the device to implement the drag event continuation according to the user's selection. For example, if the user selects the identification of the tablet, the PC can send the corresponding operation parameters to the tablet so that the tablet implements the drag event continuation. The devices connected to the PC can send an indication message to the PC after the creation of the virtual input device is completed, and the PC can obtain which devices connected to the PC have successfully created virtual input devices according to the received indication message, and display the list option based on the same. In other embodiments, the shuttle relationship can be preconfigured. If there are multiple devices connected to the PC, and some or all of the multiple devices have created virtual input devices, the device to implement the drag continuation can be determined according to the preconfigured shuttle relationship. For example, the multiple devices connected to the PC include the tablet, and the tablet has created a virtual input device, and the preconfigured shuttle relationship is that the mouse pointer slides out from the left side (or the left edge) of the display screen of the PC, and the device to implement the drag continuation is the tablet. Then, when the user presses and moves the mouse so that the mouse pointer slides over the left edge of the display screen of the PC, the PC can not only determine that the mouse shuttle starts, but also determine that the device to implement the drag continuation is the tablet. Of course, if there is only one device connected to the PC, and the device has created a virtual input device, the device to implement the drag continuation can also be determined according to the preconfigured shuttle relationship. For example, the preconfigured shuttle relationship is that the mouse pointer slides out from the left edge of the display screen of the PC, and the mouse shuttles to the tablet. However, when the user presses and moves the mouse so that the mouse pointer slides over the right edge of the display screen of the PC, it can be determined that the mouse does not shuttle to the tablet. In other embodiments, the device to implement the drag continuation can be determined by identifying the position of the device. For example, when the user presses and moves the mouse so that the mouse pointer slides over the left edge of the display screen of the PC, the position of the device around the PC can be identified by using positioning technologies such as Bluetooth, Ultra-wideband (UWB), ultrasonic waves, etc. For example, the PC identifies that the tablet is on the left side of the PC, and the device to implement the drag continuation is the tablet.

[0293] The shuttle relationship can be configured by the user in advance through a configuration file, or a configuration interface for configuring the shuttle relationship can be provided for the user, and the user can configure the shuttle relationship in advance through the configuration interface. For example, taking the configuration of the shuttle relationship between the PC and the tablet computer by the user through the interface as an example. The PC receives a user operation of opening the configuration interface and displays the configuration interface, and the configuration interface includes the identifier (such as the icon) of the PC and the identifier (such as the icon) of the tablet computer. The user can configure the shuttle relationship by dragging the two identifiers. As an example, if the user places the identifier of the tablet computer to the left of the identifier of the PC, the PC can determine that when the mouse pointer slides over the left edge of the display screen of the PC, the device that implements the drag continuation is the tablet computer. If the user places the identifier of the tablet computer to the right of the identifier of the PC, the PC can determine that when the mouse pointer slides over the right edge of the display screen of the PC, the device that implements the drag continuation is the tablet computer. When there are multiple devices, the shuttle relationship of each device can be configured in advance. The following embodiments are described by taking the tablet computer as an example. It should be noted that for the two implementations of determining the device that implements the drag continuation according to the preconfigured shuttle relationship and the recognized device position, the above S1501 can be executed before the mouse shuttle trigger or after the mouse shuttle trigger, which is not limited in this embodiment.

[0294] For example, in combination with Figure 2B After the user selects the content to be dragged by using the mouse, the PC keyboard and mouse module can receive a corresponding operation, such as a mouse operation in which the user selects the content. The user can move the mouse so that the mouse pointer of the PC is displayed on the content selected by the user. Then, if the user presses and moves the mouse (that is, uses the mouse of the PC to input a press operation (such as a mouse press operation) and a movement operation (such as a mouse movement operation)), the PC keyboard and mouse module can correspondingly receive a press event (such as a mouse press event) and a movement event (such as a mouse movement event). At this time, because the mouse shuttle condition has not been triggered, the mounted hook does not intercept the input event, and the mouse press event and the mouse movement event are transmitted to the windows system of the PC. According to the received mouse press event and the mouse movement event, the windows system of the PC can initiate a drag event for the content, and draw the content, such as an animation (or a drag animation) in which the shadow of the content moves with the mouse pointer, on the display screen of the PC. For example, as shown in Figure 16 The user wants to drag the picture 1601 of the PC to the tablet computer and continue the drag in the tablet computer. The user selects the picture 1601 by using the mouse 1602, and then presses and moves the mouse 1602. As the mouse 1602 moves, the PC correspondingly displays an animation in which the shadow 1606 of the picture 1601 moves with the mouse pointer 1604 on the display screen 1603 of the PC. Figure 16The dragging track of the shadow 1606 of the picture 1601 in the PC display screen 1603 as the mouse 1602 moves is shown as the track 1605. It should be noted that in the embodiment, the operation of selecting the content, such as the mouse operation of selecting the content in S1502, is optional. For example, when the content that the user wants to drag is a file or a folder, the mouse operation of selecting the content can not be performed, and when the mouse pointer is displayed on the file or the folder, the press operation and the moving operation can be performed, so as to initiate the dragging event of the file or the folder.

[0295] As described in S1501, after the keyboard and mouse sharing mode is started, the PC starts the input monitoring and hooks. After the dragging event is initiated, the mouse pointer moves on the PC display screen, and the keyboard and mouse module of the PC can monitor the real-time coordinate position of the mouse pointer on the PC display screen by using the input monitoring. When the keyboard and mouse module of the PC determines that the mouse pointer slides out of the edge of the PC display screen according to the monitored real-time coordinate position of the mouse pointer on the PC display screen, it is determined that the mouse shuttle condition is met. At this time, the keyboard and mouse module of the PC can determine that the mouse shuttle starts.

[0296] After the keyboard and mouse module of the PC determines that the mouse shuttle starts, the keyboard and mouse module of the PC can send the shuttle state information (the shuttle state information for indicating that the mouse starts to shuttle can be the first shuttle state information in the application) for indicating that the mouse starts to shuttle to the tablet computer through the transmission management module of the PC through the connection established between the PC and the tablet computer. After the tablet computer receives the information, the tablet computer can simulate a mouse pointer and display the mouse pointer on the display screen of the tablet computer (the mouse pointer displayed on the tablet computer can be the first cursor in the application). The keyboard and mouse module of the PC can also hide the mouse pointer displayed on the PC display screen. Of course, the object moving with the mouse pointer, such as the shadow of the object, is also hidden. For example, in combination with Figure 16 , as the mouse 1602 moves, after the mouse pointer 1604 slides over the edge of the PC display screen 1603, the PC hides the shadow 1606 of the picture 1601 and the mouse pointer 1604 displayed on the PC display screen 1603. In addition, the tablet computer displays the mouse pointer on the display screen of the tablet computer. The visual effect of the mouse pointer from the PC to the tablet computer is provided to the user.

[0297] After the PC's keyboard and mouse module determines that the mouse shuttle starts, if the user operates the mouse, the PC's keyboard and mouse module can use the HOOK to intercept the received corresponding input event, such as the mouse event, and capture the operation parameter, such as the mouse operation parameter, in the intercepted mouse event. The mouse operation parameter can include the mouse key flag bit (used to indicate which operation, such as the pressing, lifting, moving or scroll wheel scrolling, is performed by the user on the mouse), the coordinate information (used to indicate the X coordinate and Y coordinate of the mouse movement when the user moves the mouse), the scroll wheel information (used to indicate the X axis distance and Y axis distance of the scroll wheel scrolling when the user operates the scroll wheel of the mouse), and the key bit information (used to indicate which key, such as the left key, the middle key or the right key, is operated by the user on the mouse). The PC's keyboard and mouse module can also transmit the captured operation parameter, such as the mouse operation parameter, to the tablet computer through the established connection by the transmission management module of the PC, so that the tablet computer makes a corresponding response. For example, continuing to combine the example shown in Figure 16 After the mouse pointer slides over the edge of the PC display screen, the user continues to move the mouse in the same direction. The PC's keyboard and mouse module can receive the movement event, such as the mouse movement event. At this time, the PC's keyboard and mouse module can use the HOOK to intercept (or shield) it, so that the mouse movement event will not be sent to the windows system of the PC, so that the PC will not respond to the received mouse movement event. The PC's keyboard and mouse module can also capture the operation parameter, such as the mouse operation parameter, in the intercepted mouse movement event, and send the captured mouse operation parameter to the tablet computer through the established connection by the transmission management module of the PC. As an example, when the mouse event is the mouse movement event, the corresponding mouse operation parameter can be the mouse key flag bit used to indicate that the user moves the mouse, the coordinate information used to indicate the X coordinate and Y coordinate of the mouse movement, the scroll wheel information (with a value of null) and the key bit information (with a value of null).

[0298] In addition, after the mouse shuttle is started, the PC (e.g., the drag management module of the PC) can identify the current drag state of the PC (i.e., whether the PC is currently performing a drag). If the PC is currently performing a drag, the PC can initiate the continuation of the drag event, or in other words, initiate the cross-screen drag. For the windows side, since the drag event needs to be initiated from a window and also needs to be received by a window, after determining that the mouse shuttle is started, the PC can display an invisible window, or in other words, a phantom window. For example, after the mouse shuttle is started, the PC can send a callback indication of the start of the mouse shuttle to the drag management module of the PC. The drag management module of the PC can send a request for creating a phantom window to the window management module of the PC according to the callback indication. The window management module of the PC can create and display the phantom window according to the request. For example, the window management module of the PC can display the phantom window at the edge of the display screen of the PC. In this case, the transparency of the phantom window is greater than a threshold, for example, the transparency of the phantom window is very high, or completely transparent.

[0299] After the phantom window is displayed, if the mouse shuttle occurs while the PC is performing a drag, or in other words, if the windows system of the PC initiates a drag event for content when the mouse shuttle occurs, the phantom window can receive the drag event from the windows system. If the mouse shuttle occurs while the PC is not performing a drag, or in other words, if the user moves the mouse without selecting any content, the phantom window does not receive the drag event. After the phantom window receives the drag event, the window management module of the PC can obtain the drag event content from the drag event received by the phantom window. For example, the window management module of the PC can capture the drag event content from the drag event through a DragEnter event. After obtaining the drag event content, the window management module of the PC can send the drag event content to the tablet through the transmission management module of the PC through the established connection with the tablet. In this case, the PC can also perform serialization processing on the drag event content before sending the drag event content to the tablet, or in other words, the drag event content sent by the PC to the tablet can be data obtained after serialization processing.

[0300] For example, the PC can send the drag event content to the tablet through the transmission management module of the PC through the established connection with the tablet. In this case, the PC can also perform serialization processing on the drag event content before sending the drag event content to the tablet, or in other words, the drag event content sent by the PC to the tablet can be data obtained after serialization processing. Figure 17As shown, it is a data structure diagram of a drag event provided by the embodiment of the present application. When the drag event enters the invisible window, the invisible window receives the data object corresponding to the drag event, such as IDataObject. The window management module of the PC can attach it to COleDataObject. Then, the content of the drag event in the IDataObject is obtained through the DragEnter event, such as calling the GetData function. In the embodiment, the content of the drag event required for implementing the continuation of the drag event can include text or file path, etc. As an example, the window management module of the PC can obtain the text in the IDataObject through the GetData(CF_UNICODETEXT) function. The window management module of the PC can obtain the file path in the IDataObject through the GetData(CF_HDROP) function. After obtaining the text or file path, it can be sent to the tablet computer after serialization processing.

[0301] In addition, in the embodiment, in order to display the object being dragged on the tablet computer side, such as the animation of the shadow of the object moving with the mouse pointer, since the shadow and the bitmap can be converted to each other, the PC also needs to obtain the bitmap of the shadow displayed on the PC side. For example, the PC can obtain the bitmap of the shadow by intercepting the image of the content being dragged displayed on the display screen of the PC. For another example, taking the content being dragged as text as an example, the PC can generate the bitmap of the shadow according to the text being dragged. For another example, taking the content being dragged as a file as an example, the PC can find the content being dragged according to the obtained file path of the content being dragged, to determine the type of the content being dragged (such as an image file), and then can use the corresponding default material as the bitmap of the shadow according to the type, or can obtain the thumbnail of the content as the bitmap of the shadow according to the obtained file path of the content being dragged. The bitmap of the shadow can be sent to the tablet computer through the transmission management module of the PC through the established connection with the tablet computer. Wherein, the PC can also perform serialization processing on the bitmap of the shadow before sending it to the tablet computer, that is, the bitmap of the shadow sent to the tablet computer by the PC can be data obtained after serialization processing.

[0302] S1507, the tablet computer receives the drag event content and the bitmap of the shadow, and initiates the drag event according to the drag event content and the bitmap of the shadow.

[0303] S1508, the tablet computer receives the mouse operation parameter, and simulates the mouse moving event according to the mouse operation parameter.

[0304] S1509, the tablet computer generates the mouse down event.

[0305] After the tablet receives the drag event content and the shadow bitmap, it can parse them and initiate the drag event. For the Android system, since drag events need to be initiated and received by view controls, the tablet can open a transparent activity, or an invisible activity. This invisible activity has view controls with transparency greater than a threshold. Using this view control, by calling the Android Open Source Project (AOSP) interface, the tablet can initiate the corresponding drag event based on the received drag event content and shadow bitmap, thus enabling the continuation of drag events on the tablet.

[0306] For example, in combination Figure 2B ,and Figure 17 In the example shown, after the tablet's transfer management module receives the drag event content and the bitmap of the shadow, it can transmit these to the tablet's drag management module. The tablet's drag management module parses the received drag event content to obtain the text or file path from the PC. Based on the obtained text or file path, the PC's drag management module can construct the drag event content data (clipData). Additionally, the tablet's drag management module can generate the corresponding shadow based on the received shadow bitmap. Then, the tablet's drag management module can use the view control of the transparent activity opened on the tablet to call the startDragAndDrop method provided by the AOSP interface, using clipData and the shadow as input parameters, to initiate the drag event on the tablet. Of course, if the PC serializes the drag event content and shadow bitmap before sending it to the tablet, the tablet can deserialize the received data to obtain the drag event content and shadow bitmap.

[0307] For example, Figure 18 The diagram shown is a data structure diagram of a drag event (DragEvent) on an Android device according to an embodiment of this application. Combined with... Figure 18The drag management module of the tablet can construct content data (clipData) according to the received text or file path (uri) from the PC (where the text or file path is contained in the item included in the content data), and generate a corresponding shadow according to the received bitmap of the shadow, and then call the startDragAndDrop method of AOSP, with the clipData and the shadow, and other parameters (such as action (which can include start, enter, hover, release, leave, end, etc.), current x coordinate, current y coordinate, local state, clip description, etc.) obtained according to the user's operation information of the mouse (such as the received mouse movement event) as input parameters, to initiate (dispatch) the drag event (DragEvent) on the tablet side.

[0308] It should be noted that, in combination with Figure 18 The label is included in the clip description. The label is used to indicate whether the drag event is initiated by the drag management module of the tablet or by an application in the tablet. For example, the label is a string (String). For example, when the label is "windowscast", it is used to indicate that the drag event is initiated by the drag management module of the tablet; when the label is not "windowscast", it is used to indicate that the drag event is not initiated by the drag management module of the tablet, but by an application in the tablet. In the above example, since the drag event is initiated by the drag management module, the label in the clip description of the drag event is "windowscast". The specific description and construction rules of the other parameters are similar to the corresponding implementation of the prior art for generating the original drag event on the Android side, and will not be described here.

[0309] It can be understood that the execution of the drag event can be triggered by a drag operation, which can include a pressing operation (such as a mouse pressing operation) and a moving operation (such as a mouse moving operation). When the user wants to pass the content of the PC to the tablet computer by means of dragging and continue to drag in the tablet computer, after the user triggers the mouse shuttle by moving the mouse, the user will continue to move the mouse in the same direction. That is, the user will use the PC input device to input the moving operation, such as using the mouse of the PC to input the mouse moving operation. At this time, as described in S1505, after the user uses the mouse of the PC to input the mouse moving operation, the keyboard and mouse module of the PC will intercept the received moving event (such as the mouse moving event) and send the mouse moving event to the tablet computer through the transmission management module of the PC after capturing and processing the operation parameters (such as the mouse operation parameters) contained in the mouse moving event. At this time, the transmission management module of the tablet computer can receive the mouse operation parameters. Since the operating systems of the PC and the tablet computer are different, the key values of the mouse operation parameters in the mouse events are different. Therefore, after the tablet computer receives the mouse operation parameters, it can convert the key codes of the received mouse operation parameters into key codes that can be recognized by the tablet computer according to the preset mapping relationship. Then, the tablet computer (such as the keyboard and mouse module of the tablet computer) can simulate the input event (such as the mouse event) that can be recognized by the tablet computer according to the mouse operation parameters after the key code conversion, that is, can simulate the moving event (such as the mouse moving event) that can be recognized by the tablet computer. The keyboard and mouse module of the tablet computer can also send the mouse moving event to the framework layer of the tablet computer.

[0310] Furthermore, at the start of mouse navigation, the initiating end, i.e., the PC, is in a drag-and-drop state, meaning the PC is in a mouse-pressed state. The target end, i.e., the tablet, can only receive mouse movement events at this time, meaning it is not in a mouse-pressed state. Therefore, the tablet can generate a press operation, such as a mouse press operation. For example, after the tablet calls the `startDragAndDrop` method provided by the AOSP interface, it receives a drag-and-drop start callback (`onDragStart`). At this point, the tablet can determine whether the drag-and-drop event was initiated by its drag-and-drop management module based on the label in the drag-and-drop event. When it is determined that the drag-and-drop event was initiated by the tablet's drag-and-drop management module (e.g., label "windowscast"), the tablet can generate (or simulate) a press event, such as a mouse press event, using the created virtual input device, based on the operation parameters of the press operation, such as a mouse press operation. For example, the tablet's drag-and-drop management module controls the tablet's keyboard and mouse module to send a mouse press event to the tablet's frame layer using the created virtual input device. In this way, the drag-and-drop event initiated in S1507 on the tablet can be attached to the mouse pointer displayed on the tablet's screen.

[0311] S1510, the tablet computer executes drag events based on mouse movement events and mouse press events, and displays an animation of the shadow of the content moving with the mouse pointer on the tablet computer's screen.

[0312] The tablet computer (such as the tablet's frame layer) can execute the drag event initiated in S1507 based on movement events, such as mouse movement events and press events, such as mouse press events. During the execution of the drag event, the tablet computer can also display content on its screen, such as an animation of the content's shadow moving with the mouse pointer (which can be the first cursor in this application). For example, combined with... Figure 16 ,like Figure 19 As shown, as the mouse 1901 moves, the shadow 1903 of the corresponding image displayed on the tablet computer's screen 1902 moves with the mouse pointer 1904 in an animation. Figure 19 The shadow 1903 in the image shown is dragged along the mouse pointer 1904 as shown in trajectory 1905.

[0313] In the case that the user drags the content from the PC to the tablet by dragging and continues to drag in the tablet, since the drag event is attached to the mouse pointer displayed on the tablet display screen, the user can accurately select the mouse release point by using the mouse pointer. For example, if the user wants to use or process the content in the tablet, the user can release the mouse after moving the mouse pointer to the view control of the tablet where the user wants to use or process the content. After the mouse is released, the PC keyboard and mouse module can receive the corresponding lifting event, such as a mouse lifting event. Since the mouse lifting event is received by the PC after the mouse shuttle starts, the PC keyboard and mouse module uses the hook to intercept (or shield) the mouse lifting event so that the mouse lifting event is not sent to the PC windows system, thereby preventing the PC from responding to the received mouse lifting event. The PC keyboard and mouse module can also use the hook to capture the operation parameters of the mouse lifting event, such as the mouse operation parameters, and send the captured mouse operation parameters to the tablet through the PC transmission management module and the established connection. The mouse operation parameters of the mouse lifting event can include a mouse key flag bit indicating that the user has lifted the mouse, coordinate information (value is empty), scroll wheel information (value is empty), and key information indicating that the user has operated the left key of the mouse. Correspondingly, the tablet transmission management module can receive the mouse operation parameters. After the tablet receives the mouse operation parameters, the tablet can convert the key code of the received mouse operation parameters into a key code recognizable by the tablet according to a preset mapping relationship. Then, the tablet can simulate a mouse event recognizable by the tablet according to the converted operation parameters, such as the mouse operation parameters, by using the created virtual input device, that is, can simulate a corresponding input event, such as a mouse lifting event.

[0314] After simulating the mouse up event, the tablet can determine the mouse release point according to the current coordinate position of the mouse pointer. For example, after the tablet learns that the key mouse shuttle starts, the tablet can register a listener of the coordinate position of the mouse pointer. In this way, the tablet can monitor the coordinate position of the mouse pointer on the display screen of the tablet in real time through the listener. Based on this, after the tablet receives the mouse up event, the tablet can obtain the current coordinate position of the mouse pointer on the display screen of the tablet by using the listener. According to the obtained coordinate position, the tablet can determine the mouse release point. For another example, after the tablet initiates the drag event by calling the startDragAndDrop method of AOSP, the tablet monitors the input event. For example, when the user continues to drag in the tablet, the tablet can monitor the movement event, such as the mouse movement event, and obtain the operation parameter of the mouse movement event, such as the mouse operation parameter, from the MotionEvent. The mouse operation parameter includes coordinate information indicating the mouse position. Then, after the user releases the finger, the tablet monitors the up event, such as the mouse up event, and determines the coordinate position of the mouse pointer according to the previously obtained coordinate information, so as to determine the mouse release point according to the obtained coordinate position.

[0315] After the user releases the mouse, since the PC side directly intercepts the received up event, such as the mouse up event, the drag event of the PC side is not released. At this time, the tablet can send a signaling to the PC to inform the PC to release the drag event after receiving the mouse up event through the established connection.

[0316] In addition, the tablet computer can also respond to a lift event, such as a mouse lift event. For example, taking the text in S1502 as an example, after simulating a mouse lift event, the drag management module of the tablet computer can send the content data in the above-mentioned drag event to the view control of the mouse release point, and the content data includes the text. After receiving the content data, the view control can perform corresponding processing according to the text in the content data, such as displaying the text in the view control. For another example, taking the file in S1502 as an example. Since the file is not actually transmitted to the tablet computer during the process of dragging the file on the tablet computer, after the user releases the mouse and the PC receives the above-mentioned signaling from the tablet computer indicating the release of the drag event, the PC can transmit the file to the tablet computer. After receiving the file, the tablet computer can store the file in a predetermined cache directory, and at this time, the drag management module of the tablet computer can obtain the uri (such as uri 1) of the file, which is the path of the file in the cache directory, and is different from the uri in the content of the drag event sent by the PC to the tablet computer in S1507 (the uri sent by the PC is the storage path of the file on the PC). The drag management module of the tablet computer can construct new content data according to the uri 1, and send the content data to the view control of the mouse release point as a response to the mouse lift event. After receiving the content data, the view control can perform corresponding processing, such as displaying the file if the view control is a view control in a memo, or sending the file if the view control is an input box in a chat window.

[0317] In some other embodiments, when the tablet computer also establishes a connection with another device, such as a mobile phone, if the user wants to continue to drag the content to the mobile phone, the user can continue to move the mouse to make the mouse pointer on the tablet computer slide over the edge of the display screen of the tablet computer, triggering the mouse to shuttle from the tablet computer to the mobile phone, so as to continue to implement the continuation of the drag event on the mobile phone. It should be noted that after the mouse is shuttled to the mobile phone, the PC can directly interact with the mobile phone to implement the continuation of the drag event on the mobile phone. The specific description of implementing the continuation of the drag event on the mobile phone is similar to that of the tablet computer, and will not be repeated here.

[0318] It can be understood that, through the above process, without starting the screen projection, the user can use the input device, such as the mouse of the PC, to move the content, such as the text or the file in the PC application, to the edge of the PC screen in a dragging manner and continue to move in the same direction, so as to trigger the mouse shuttle. After the mouse shuttle starts, the mouse pointer appears on the tablet computer. And the PC sends the dragging event content to the tablet computer, so that the dragging event can be continued to be attached to the mouse pointer of the tablet computer, to realize the continuation of the dragging event on the tablet computer, and give the user a visual effect that the content is dragged from the PC to the tablet computer.

[0319] In the embodiment of the present application, the user can not only have the requirement of transferring the content of the PC to the tablet computer in a dragging manner and continuing to drag in the tablet computer, but also can have the requirement of transferring the content of the tablet computer to the PC in a dragging manner. For example, in combination with the example in S1501-S1510, the user transfers the picture 1 of the PC to the tablet computer in a dragging manner, and after continuing to drag in the tablet computer, releases the mouse. Then, the user edits the picture 1 by using the handwriting pen of the tablet computer and saves it. The user wants to transfer the edited picture 1 to the PC in a dragging manner, and after continuing to drag in the PC, releases the mouse, so as to save the edited picture 1 on the PC.

[0320] Please refer to Figure 20 Another flowchart of the object dragging method across devices provided by the embodiment of the present application is provided. The following introduces the method provided by the embodiment of the present application, taking the example that after the mouse shuttle occurs, the user wants to transfer the content of the tablet computer to the PC in a dragging manner and continue to drag in the PC. Figure 15 As shown in the figure, the method can include the following S2001-S2010. Figure 20

[0321] S2001, the tablet computer receives the mouse operation of the selected content from the PC.

[0322] S2002, the tablet computer receives the mouse operation parameters of the mouse down event and the mouse move event from the PC, simulates the mouse down event and the mouse move event according to the mouse operation parameters, initiates the dragging event according to the mouse down event and the mouse move event, and displays the animation that the shadow of the content moves with the mouse pointer on the display screen of the tablet computer.

[0323] In the embodiment, the mouse operation parameters of the mouse down event can be the first operation parameter in the present application, and the mouse operation parameters of the mouse move event can be the second operation parameter in the present application.

[0324] As shown in the figure, the method can include the following S2001-S2010. Figure 15 ​In the illustrated embodiment, after the mouse shuttle occurs, the PC will intercept the corresponding input event, such as the mouse event, by using the mounting HOOK behind the user operating the input device of the PC, such as the mouse, and the PC will capture the operation parameters, such as the mouse operation parameters, contained in the intercepted mouse event and send them to the tablet computer. For example, after the mouse shuttle to the tablet computer, the user wants to transfer the content in the tablet computer to the PC by dragging and continue to drag in the PC. In combination with Figure 2B , the user selects the content to be dragged by using the input device of the PC, such as the mouse. The PC's keyboard and mouse module can receive the corresponding input operation, such as the mouse operation. The PC's keyboard and mouse module will intercept the received corresponding input event, such as the mouse event, by using the HOOK, so that the input event will not be sent to the PC's windows system, so that the PC will not respond to the received input event. The PC's keyboard and mouse module will also capture the operation parameters, such as the mouse operation parameters, in the intercepted input event by using the HOOK, and transmit them to the tablet computer through the PC's transmission management module through the established connection. At this time the transmission management module of the tablet computer can receive the corresponding operation parameters, such as the mouse operation parameters. The tablet computer can simulate the corresponding input event, such as the mouse event, by using the created virtual input device after the received operation parameters are converted into key codes.

[0325] After the mouse pointer of the tablet PC displays on the object to be dragged, such as the selected content, the user can use the input device of the PC, such as the mouse, to input the drag operation, so that the tablet PC can implement the drag operation on the corresponding object, i.e. the content, according to the drag operation. The drag operation can be an operation or a plurality of operations. For example, the drag operation includes two operations, i.e. the pressing operation and the moving operation. Taking the mouse as an example, the pressing operation can be the mouse pressing operation and the moving operation can be the mouse moving operation. The user can press and move the mouse of the PC, i.e. use the mouse of the PC to input the mouse pressing operation and the mouse moving operation. Then, the keyboard and mouse module of the PC can receive the pressing event (such as the mouse pressing event) and the moving event (such as the mouse moving event). Similarly, the keyboard and mouse module of the PC can use the HOOK to intercept the received mouse pressing event and mouse moving event, and use the HOOK to capture the operation parameters of the intercepted mouse pressing event and mouse moving event, such as the mouse operation parameters, and then transmit the captured operation parameters to the tablet PC through the transmission management module of the PC and the established connection. At this time, the transmission management module of the tablet PC can receive the corresponding operation parameters, such as the mouse operation parameters. After the tablet PC converts the received operation parameters into the key code, the tablet PC can simulate the corresponding input events, such as the pressing event (such as the mouse pressing event) and the moving event (such as the mouse moving event), using the created virtual input device.

[0326] After the tablet PC simulates the pressing event (such as the mouse pressing event) and the moving event (such as the mouse moving event), the tablet PC can initiate the drag event in response to the events and draw the content, such as the shadow of the content, moving with the mouse pointer, on the display screen of the tablet PC. For example, the user uses the stylus of the tablet PC to edit the picture 1 dragged from the PC and saves it. The user wants to drag the edited picture to the PC and continue to drag it on the PC. As shown in FIG. 21, the user can use the mouse 2101 to select the edited picture 2102, and then press and move the mouse 2101. With the movement of the mouse 2101, the tablet PC displays the shadow 2106 of the edited picture 2102 moving with the mouse pointer 2104 on the display screen 2103 of the tablet PC, as shown in FIG. 22. Figure 21 Figure 21

[0327] ​​It should be noted that the above is described by taking the event triggering the dragging as an example, which includes the pressing event and the moving event. For example, the user can trigger the dragging by pressing and moving the mouse. In some other embodiments, the event triggering the dragging can only include the pressing event. For example, the user can trigger the dragging by long-pressing the mouse, or the user can trigger the dragging by pressing the mouse. The embodiments are not limited specifically herein.

[0328] S2003, the tablet computer obtains the bitmap of the dragging event content and the shadow, and caches the obtained bitmap of the dragging event content and the shadow.

[0329] The dragging event content is used for the dragging continuation end device to construct the dragging event. For example, when the content in S2001 is text, the dragging event content can include the text (text). For another example, when the content in S2001 is a file or a folder, the dragging event content can include the file path (such as uri).

[0330] For example, in combination with the description of the embodiment shown in Figure 2B When the application of the tablet computer initiates the dragging event, the corresponding dragging event content can be called back to the application layer by the framework layer. For example, when the application initiates the dragging event by calling the startDragAndDrop method provided by the AOSP interface, the framework layer can extract the dragging event content by means of program insertion. For example, the location and content that need to be inserted can be determined in advance, and then the program is inserted according to the determined location and content, so that the dragging event content can be extracted. Then, the framework layer of the tablet computer can call back the extracted dragging event content to the dragging management module of the application layer of the tablet computer. After obtaining the dragging event content, the dragging management module of the tablet computer can cache it. As an example, the dragging event content required for implementing the continuation of the dragging event can include text or file path. When the application initiates the dragging event, the framework layer can extract the text or file path by means of program insertion.

[0331] In addition, in the embodiment, the dragging event can have two initiation modes. One is initiated by the dragging management module of the tablet computer (that is, the dragging is from the PC to the tablet computer, such as the description in the embodiment shown in Figure 15 In combination with the description of the embodiment shown in Figure 15 Figure 18 ​It can be understood that the content of the drag event needs to be acquired and cached only when the drag event is initiated by the application of the tablet computer or is not initiated by the drag management module of the tablet computer, so as to be used for the PC to continue the drag event after the drag shuttle to the PC. That is, after the drag event is initiated, the tablet computer can distinguish whether the drag event is initiated by the application of the tablet computer or by the drag management module of the tablet computer according to the label in the content description of the drag event. For example, when the label is not "windowscast", it can be determined that the drag event is not initiated by the drag management module of the tablet computer, but is initiated by the application of the tablet computer. When the drag event is not initiated by the drag management module of the tablet computer, the framework layer needs to acquire the content of the drag event and send the acquired content of the drag event to the drag management module of the tablet computer, so as to cache the content of the drag event.

[0332] In addition, in the embodiment, the framework layer can also acquire the bitmap of the shadow by adding an interface or using an original interface (such as using the interface of callback clipData). The acquired bitmap of the shadow can also be called back to the drag management module of the application layer of the tablet computer. After the drag management module of the tablet computer acquires the bitmap of the shadow, the bitmap of the shadow can also be cached.

[0333] S2004, the tablet computer monitors the coordinate position of the mouse pointer on the display screen of the tablet computer.

[0334] S2005, the tablet computer sends the shuttle state information for indicating the end of the mouse shuttle to the PC according to the coordinate position of the mouse pointer on the display screen of the tablet computer when it is determined that the mouse pointer slides out of the edge of the display screen of the tablet computer.

[0335] S2006, the PC receives the shuttle state information for indicating the end of the mouse shuttle.

[0336] The shuttle state information for indicating the end of the mouse shuttle can be the second shuttle state information in the application.

[0337] In the process that the content (e.g. the shadow of the content) follows the movement of the mouse pointer, the tablet can determine whether the dragged content (e.g. the shadow of the content) is dragged out of the edge of the display screen of the tablet. When the content (e.g. the shadow of the content) is dragged out of the edge of the display screen of the tablet, it indicates that the user wants to use the mouse to control other devices. In the embodiment, the specific description of the content dragged out of the edge of the display screen of the tablet is similar to that of the content dragged out of the PC, which will not be described in detail here. The following takes the specific example of determining whether the mouse pointer is dragged out of the edge of the display screen of the tablet as an example.

[0338] As an example, after the drag event is initiated, the mouse pointer moves on the display screen of the tablet, and the tablet can monitor the real-time coordinate position of the mouse pointer on the display screen of the tablet. For example, after the mouse and keyboard switching is started, the tablet can register a listener of the coordinate position of the mouse pointer. In this way, the tablet can monitor the coordinate position of the mouse pointer on the display screen of the tablet in real time through the listener. The tablet can determine whether the mouse pointer slides out of the edge of the display screen of the tablet according to the real-time coordinate position of the mouse pointer on the display screen of the tablet monitored by the listener. For example, the tablet can determine the coordinate position of the mouse pointer on the display screen of the tablet according to the initial position and the relative displacement of the mouse pointer, so as to determine whether the mouse pointer slides out of the edge of the display screen of the tablet. The initial position of the mouse pointer can be the coordinate position of the mouse pointer on the display screen of the tablet when the mouse starts to move, or the coordinate position of the mouse pointer on the display screen of the tablet before the mouse starts to move. The initial position of the mouse pointer can be the coordinate position in the coordinate system with the upper left corner of the display screen of the tablet as the coordinate origin, the X axis pointing to the right edge of the display screen of the tablet from the upper left corner, and the Y axis pointing to the lower edge of the display screen of the tablet from the upper left corner. The specific implementation of the tablet to determine whether the mouse pointer slides out of the edge of the display screen of the tablet is similar to that of the PC to determine whether the mouse pointer slides out of the edge of the display screen of the PC, which will not be described in detail here.

[0339] After the tablet determines that the mouse pointer slides over the edge of the tablet display screen, it indicates that the user wants to use the mouse to control other devices. As described in S1506, if the tablet only establishes a connection with one device, i.e. the PC, it indicates that the user wants to use the mouse to control the PC. If the tablet establishes a connection with multiple devices, the mobile phone can display a list option including the identification of all devices connected to the tablet, for the user to select the device that the user wants to control with the mouse. If the user selects the identification of the PC, it indicates that the user wants to use the mouse to control the PC. Alternatively, the tablet can be pre-configured with a shuttle relationship for determining to which device the mouse is shuttled, i.e. which device is to respond to the operation of the mouse. The configuration and application of the shuttle relationship are similar to the corresponding content described in the above embodiments, and will not be described in detail here. After determining that the user wants to use the mouse to control the PC, the tablet can determine that the mouse shuttle ends. At this time, the tablet can send the shuttle state information indicating that the mouse shuttle ends to the PC through the established connection.

[0340] S2007, the PC determines that the mouse shuttle ends according to the received shuttle state information.

[0341] S2008, the PC cancels the interception of the mouse event and generates a mouse down event.

[0342] S2009, the PC obtains the drag event content and the bitmap of the shadow from the tablet, and initiates a drag event according to the drag event content and the bitmap of the shadow.

[0343] In this application, the drag data can include the drag event content and the bitmap of the shadow. After the PC receives the shuttle state information indicating that the mouse shuttle ends, it can determine that the mouse shuttle ends.

[0344] After determining that the mouse shuttle ends, the PC can display the mouse pointer on the PC display screen. In combination with the description in the embodiments shown in Figure 15 In this embodiment, when the mouse shuttle ends, the PC cancels the hiding of the mouse pointer, and the mouse pointer is displayed on the edge of the PC display screen. Of course, when the mouse shuttle ends, the mouse pointer on the tablet side will also not be displayed. This gives the user a visual effect of the mouse pointer being shuttled from the tablet to the PC.

[0345] After determining the end of the mouse shuttle, the PC also needs to uninstall the HOOK (or close the HOOK) to cancel the interception of the input device, such as the mouse event. For example, after the user triggers the end of the mouse shuttle by moving the mouse to transfer the content of the tablet computer to the PC by dragging, the user continues to move the mouse in the same direction, and the PC keyboard and mouse module can receive a movement event, such as a mouse movement event. Since the HOOK has been uninstalled at this time, the PC keyboard and mouse module will send the received movement event, such as the mouse movement event, to the PC windows system, so that the PC windows system responds to the movement event.

[0346] In addition, before the end of the mouse shuttle, the tablet computer is in a state of dragging, and the mouse shuttle back to the PC is to realize the continuation of the dragging, that is, the PC needs to continue to realize the dragging. Since the Windows platform must press the mouse on the window to normally initiate the dragging. But in the process of the user continuing to move the mouse, the PC can only receive a movement event, such as a mouse movement event, that is, it will not receive a press event, such as a mouse press event. Therefore, the PC (such as the dragging management module of the PC) can generate a press event, such as a mouse press event, and transmit it to the invisible window. For example, after the PC determines the end of the mouse shuttle, the tablet computer is in the process of dragging, the PC can generate a mouse press event and transmit it to the invisible window, so that the dragging event initiated in the PC end S2009 can be attached to the mouse pointer displayed on the PC display screen, that is, the continuation of the dragging event is realized on the PC end.

[0347] In this embodiment, after determining the end of the mouse shuttle, the PC can also request the tablet computer for the dragging state of the tablet computer (that is, whether it is in the process of dragging), and when the dragging state returned by the tablet computer indicates that it is in the process of dragging, the PC can request the tablet computer for the dragging event content and the bitmap of the shadow. The PC can send a request message to the tablet computer, which can be used to request the dragging data, that is, to request the dragging event content and the bitmap of the shadow.

[0348] For example, in combination with the above-mentioned embodiment, the PC can determine the end of the mouse shuttle by detecting the mouse press event, and the PC can also determine the end of the mouse shuttle by detecting the mouse release event. Figure 2BWhen the PC determines that the mouse shuttle ends, the PC's keyboard and mouse module can send an indication of the end of the mouse shuttle to the PC's drag management module. According to the received indication, the PC's drag management module can request the drag state from the tablet through the PC's transmission management module. When the tablet returns a drag state indicating that the tablet is in a drag, the PC's drag management module can request the drag event content and the bitmap of the shadow from the tablet through the PC's transmission management module. Correspondingly, the tablet's transmission management module can receive the request and forward the request to the tablet's drag management module. After receiving the request, the tablet's drag management module can feed back the bitmap of the drag event content and the shadow cached in S2003 to the PC's transmission management module through the tablet's transmission management module.

[0349] According to the description in S2003, after receiving the drag event content, the PC's transmission management module can transmit the drag event content to the PC's drag management module. The PC's drag management module can analyze the received drag event content and obtain the text or file path from the tablet. According to the obtained text or file path, the PC's drag management module can construct a data object of the drag event, such as IDataObject. In addition, after receiving the bitmap of the shadow, the PC's transmission management module can restore the shadow on the PC according to the bitmap. For example, the IDragSourceHelper interface provided by the PC can be used to restore the shadow. After that, the PC can initiate the drag event on the PC. For example, according to the description of the embodiment shown in Figure 15 According to the description of the embodiment shown in

[0350] S2010, the PC executes the drag event according to the mouse movement event and the mouse press event, and displays an animation of the shadow of the content moving with the mouse pointer on the display screen of the PC.

[0351] After the drag event is initiated, the PC can execute the drag event in response to the movement event, such as the mouse movement event, and the press event, such as the mouse press event. And the PC can display an animation of the shadow of the content, such as the shadow of the content, moving with the mouse pointer on the display screen of the PC. For example, as shown in Figure 21 For example, as shown in Figure 22 As the mouse 2201 moves, the PC displays an animation of the shadow 2203 of the edited picture moving with the mouse pointer 2204 on the display screen 2202 of the PC, as shown in Figure 22 The drag track of the shadow 2203 moving with the mouse pointer 2204 is shown as track 2205.

[0352] It should be noted that the above examples are described by taking the user first dragging the content from the PC to the tablet and then dragging the content from the tablet to the PC as an example. In some other embodiments, the user can not drag the content from the PC to the tablet, but directly drag the content of the tablet to the PC. The specific implementation of such embodiments is similar to that of the above examples. Figure 15 and Figure 20 The specific implementation of the above examples is similar to that of the above examples, except that S1502 is not performed. S1503 is replaced by receiving a movement event (such as a mouse movement event) and displaying an animation of the movement of the mouse pointer on the display screen of the PC according to the movement event. S1506 is not performed, but the operation of the PC displaying the invisible window needs to be performed, except that the invisible window will not receive the drag event. S1507 and S1509 are not performed. S1510 is replaced by the tablet displaying an animation of the movement of the mouse pointer on the display screen of the tablet according to the movement event (such as a mouse movement event). That is, after the mouse shuttle starts, the tablet can receive the operation of moving the mouse pointer (referred to as a first cursor) displayed on the tablet by the user using the input device of the PC, such as a mouse. In response to the operation, the tablet can display an animation of the movement of the mouse pointer on the display screen of the tablet. If the user wants to drag a certain content, the user can continue to move the mouse of the PC until the mouse pointer moves to the content. Wherein, the tablet receives the operation of moving the first cursor by the user using the input device of the PC, such as a mouse, which can be that the PC intercepts the corresponding movement event, such as a mouse movement event, and sends the operation parameters included in the movement event, such as a third operation parameter, to the tablet. The tablet can simulate the movement event, such as a mouse movement event, according to the third operation parameter, so as to display an animation of the movement of the mouse pointer on the display screen of the tablet. Other operations are similar, and the embodiments of the present application will not be described in detail here.

[0353] The above examples are described by taking the tablet caching the drag data at the start of the drag and the PC requesting the drag data from the tablet after the end of the mouse shuttle as an example. In some other embodiments, the tablet can not cache the drag data at the start of the drag, but can obtain and send the drag data sent by the PC actively after determining the end of the mouse shuttle without the PC requesting.

[0354] The method provided in the embodiment enables a user to use an input device such as a mouse to move content such as text or a file in a plurality of terminals participating in collaborative use by dragging the content in the manner of the mouse pointer, without starting screen projection. The user can use the terminals to process the delivered content, so that the hardware capabilities of the plurality of terminals can be used in collaborative work. In addition, since screen projection is not started, the display space of the display screen of a terminal is not occupied. The use efficiency of the plurality of terminals in collaborative use is improved, and the user experience is improved.

[0355] In addition, the above embodiment is described by taking the input device as a mouse. In the embodiment, the input device can also be a touchpad. When the input device is a touchpad, the user can use the keys (left key or right key) of the touchpad to input a pressing operation, and use a finger to slide on the touchpad to input a moving operation. The user uses the touchpad to achieve object dragging in a manner similar to that of using the mouse to achieve object dragging, which is not described herein again.

[0356] Figure 23 A composition diagram of a cross-device object dragging apparatus provided in the embodiment is shown in FIG. 23. Figure 23 As shown in the figure, the apparatus can be applied to a second terminal (such as a mobile phone or a tablet computer), the second terminal is connected with a first terminal, and the apparatus can include a display unit 2301, a receiving unit 2302, and a sending unit 2303.

[0357] The display unit 2301 is configured to display a first cursor on an object displayed on the second terminal.

[0358] The receiving unit 2302 is configured to receive a dragging operation input by a user using an input device of the first terminal, the dragging operation being used to initiate dragging of the object.

[0359] The display unit 2301 is further configured to display an animation of the object moving with the first cursor on the display screen of the second terminal in response to the dragging operation.

[0360] The sending unit 2303 is configured to send dragging data to the first terminal when it is determined that the object is dragged out of the edge of the display screen of the second terminal.

[0361] Further, the dragging data can be used by the first terminal to display an animation of the object moving with a second cursor on the display screen of the first terminal.

[0362] Further, the object is an icon of an application; or the object is a window including an interface of the application; when the object is the icon of the application, the dragging data includes the icon of the application; and when the object is the window, the dragging data includes the interface of the application.

[0363] Further, the receiving unit 2302 is further configured to receive an indication message from the first terminal, the indication message being sent by the first terminal after the user inputs a drag release operation using an input device of the first terminal, and being used to indicate that the dragging of the object is ended.

[0364] The sending unit 2303 is further configured to send data to the first terminal, the data being used by the first terminal to display an interface of an application on a display screen of the first terminal.

[0365] Further, when the object is an icon of the application, the apparatus can further include a creating unit 2304.

[0366] The creating unit 2304 is configured to create a drag object according to the icon of the application, the drag object including an icon same as the icon of the application.

[0367] The display unit 2301 is specifically configured to display an animation of the drag object moving with the first cursor in response to the drag operation.

[0368] Further, when the object is a window, the display unit 2301 is further configured to display the interface of the application, and the receiving unit 2302 is further configured to receive a long press operation input by the user using the input device of the first terminal when the first cursor is displayed on the status bar of the second terminal, the long press operation being used to trigger the windowing of the interface of the application, and the display unit 2301 is further configured to display the window in response to the long press operation, the window specifically including a screenshot of the interface of the application, and the drag data specifically including the screenshot of the interface of the application.

[0369] Further, the display unit 2301 is specifically configured to display an animation of a shadow of the object moving with the first cursor on the display screen of the second terminal.

[0370] Further, the object is a text, a file or a folder, and the drag data includes a bitmap of the drag event content and the shadow, wherein when the object is the text, the drag event content includes the text, and when the object is the file or the folder, the drag event content is a file path.

[0371] Further, the apparatus can further include an obtaining unit 2305 configured to obtain and cache the drag data.

[0372] Further, the receiving unit 2302 is further configured to receive a request message from the first terminal, the request message being used to request the drag data, and the sending unit 2303 is specifically configured to send the cached drag data to the first terminal.

[0373] Further, the receiving unit 2302 is further configured to receive first shuttle state information from the first terminal, the first shuttle state information being used to indicate that the shuttle is started.

[0374] The display unit 2301 is further configured to display the first cursor on the second terminal in response to the first shuttle state information.

[0375] Further, the receiving unit 2302 is further configured to receive an operation of moving the first cursor input by a user using an input device of the first terminal.

[0376] The display unit 2301 is further configured to display an animation of movement of the first cursor on the display screen of the second terminal in response to the operation of moving the first cursor until the first cursor moves onto the object.

[0377] Further, the drag operation includes a pressing operation and a moving operation; the receiving unit 2302 is specifically configured to receive a first operation parameter from the first terminal, the first operation parameter being an operation parameter contained in a pressing event received by the first terminal after the user performs the pressing operation using the input device of the first terminal; and receive a second operation parameter from the first terminal, the second operation parameter being an operation parameter contained in a moving event received by the first terminal after the user performs the moving operation using the input device of the first terminal.

[0378] The apparatus can further include an emulating unit 2306 configured to emulate the pressing event according to the first operation parameter and emulate the moving event according to the second operation parameter, the pressing event and the moving event being used to instruct to initiate the drag on the object.

[0379] The display unit 2301 is specifically configured to display an animation of movement of the object along with the first cursor in response to the pressing event and the moving event.

[0380] Further, the receiving unit 2302 is specifically configured to receive a third operation parameter from the first terminal, the third operation parameter being an operation parameter contained in the moving event received by the first terminal after the user performs the moving operation using the input device of the first terminal.

[0381] The emulating unit 2306 is configured to emulate the moving event according to the third operation parameter, the moving event being used to instruct to move the first cursor.

[0382] Further, the sending unit 2303 is further configured to send second shuttle state information to the first terminal, the second shuttle state information being used to instruct to end the shuttle.

[0383] Further, the long press operation comprises a press operation and a lift operation; the receiving unit 2302 is specifically configured to receive a fourth operation parameter from the first terminal, the fourth operation parameter being an operation parameter contained in a press event received by the first terminal after a user performs the press operation using an input device of the first terminal when the first cursor is displayed on the status bar; and receive a fifth operation parameter from the first terminal, the fifth operation parameter being an operation parameter contained in a lift event received by the first terminal after the user performs the lift operation using the input device of the first terminal after performing the press operation for a period of time.

[0384] The simulation unit 2305 is configured to simulate the press event according to the fourth operation parameter and simulate the lift event according to the fifth operation parameter, and the press event and the lift event are used to trigger the interface windowing of the application.

[0385] The display unit 2301 is specifically configured to display the window in response to the press event and the lift event.

[0386] Further, the creating unit 2304 is further configured to create a virtual input device after the connection with the first terminal is successfully established; or, the receiving unit 2302 is further configured to receive a notification message from the first terminal, the notification message being used to indicate that a keyboard-mouse sharing mode of the first terminal has been started, and the creating unit 2304 is further configured to create the virtual input device in response to the notification message; wherein the virtual input device is used by the second terminal to simulate the input event according to the operation parameter.

[0387] Figure 24 Another composition diagram of the cross-device object dragging device provided by the embodiment of the present application is provided. As shown in the figure, Figure 24 the device can be applied to a first terminal (such as the PC described above), the first terminal is connected with a second terminal, and the device can comprise a sending unit 2401 and a receiving unit 2402.

[0388] The sending unit 2401 is configured to send, to the second terminal, data of a drag operation input by a user using an input device of the first terminal, the drag operation being used to initiate a drag on an object displayed by the second terminal.

[0389] The receiving unit 2402 is configured to receive, from the second terminal, drag data, the drag data being sent by the second terminal to the first terminal after the second terminal determines that the object is dragged out of the edge of the display screen of the second terminal during the movement of the object along with the second cursor on the display screen of the second terminal.

[0390] Further, the device can further comprise a display unit 2403 configured to display, on the display screen of the first terminal, an animation of the movement of the object along with the second cursor according to the drag data and the drag operation.

[0391] Further, the object is an icon of the application; or the object is a window including the interface of the application; when the object is the icon of the application, the drag data includes the icon of the application; when the object is the window, the drag data includes the interface of the application.

[0392] Further, the receiving unit 2402 is further configured to receive a drag release operation input by a user using an input device of the first terminal.

[0393] The display unit 2403 is further configured to display the interface of the application on the display screen of the first terminal in response to the drag release operation.

[0394] Further, the sending unit 2401 is further configured to send an indication message to the second terminal, the indication message being used to indicate that the dragging of the object is ended.

[0395] The receiving unit 2402 is further configured to receive data from the second terminal.

[0396] The display unit 2403 is specifically configured to display the interface of the application on the display screen of the first terminal according to the data.

[0397] Further, when the object is the window, the sending unit 2401 is further configured to send data of a long press operation input by the user using the input device of the first terminal to the second terminal, the long press operation being input by the user when the first cursor is displayed on the status bar of the second terminal and being used to trigger the windowing of the interface of the application displayed by the second terminal; the window specifically includes a screenshot of the interface of the application, and the drag data specifically includes the screenshot of the interface of the application.

[0398] Further, the display unit 2403 is specifically configured to display an animation of a shadow of the object moving with the second cursor on the display screen of the first terminal according to the drag data and the drag operation.

[0399] Further, the object is text, a file or a folder; the drag data includes a bitmap of the drag event content and the shadow; when the object is the text, the drag event content includes the text; when the object is the file or the folder, the drag event content is a file path.

[0400] Further, the apparatus can further include a determination unit 2404 configured to determine that the second cursor slides out of an edge of the display screen of the first terminal.

[0401] Further, the receiving unit 2402 is further configured to receive an operation of moving the second cursor input by the user using the input device of the first terminal.

[0402] The display unit 2403 is further configured to display an animation of the second cursor moving on the display screen of the first terminal in response to the operation of moving the second cursor.

[0403] Further, the sending unit 2401 is further configured to send first shuttle state information to the second terminal, the first shuttle state information being used to indicate a start of the shuttle.

[0404] Further, the sending unit 2401 is further configured to send, to the second terminal, data of an operation of moving the first cursor input by a user using an input device of the first terminal, so that the second terminal displays an animation of the movement of the first cursor on a display screen of the second terminal.

[0405] Further, the drag operation includes a pressing operation and a moving operation; and the apparatus can further include an intercepting unit 2405.

[0406] The intercepting unit 2405 is configured to intercept a pressing event after the user performs the pressing operation using the input device of the first terminal.

[0407] The sending unit 2401 is specifically configured to send, to the second terminal, a first operation parameter included in the pressing event.

[0408] The intercepting unit 2405 is further configured to intercept a moving event after the user performs the moving operation using the input device of the first terminal.

[0409] The sending unit 2401 is specifically configured to send, to the second terminal, a second operation parameter included in the moving event.

[0410] The first operation parameter and the second operation parameter are respectively used by the second terminal to simulate the pressing event and the moving event, and are further used to instruct to initiate the drag on the object.

[0411] Further, the intercepting unit 2405 is configured to intercept the moving event after the user performs the moving operation using the input device of the first terminal.

[0412] The sending unit 2401 is specifically configured to send, to the second terminal, a third operation parameter included in the moving event, the third operation parameter being used by the second terminal to simulate the moving event, and being further used to instruct to move the first cursor.

[0413] Further, the receiving unit 2402 is further configured to receive second shuttle state information from the second terminal, the second shuttle state information being used to indicate an end of the shuttle.

[0414] Further, the sending unit 2401 is further configured to send, to the second terminal, a request message, the request message being used to request the drag data.

[0415] Further, the apparatus can further include an input unit 2406.

[0416] The intercepting unit 2405 is configured to cancel the interception of the input event.

[0417] The input unit 2406 is configured to receive the movement event after the user performs the movement operation by using the input device of the first terminal.

[0418] The input unit 2406 is further configured to generate the press-down event.

[0419] The display unit 2403 is specifically configured to display the animation of the movement of the object with the second cursor on the display screen of the first terminal according to the movement event, the press-down event and the drag data.

[0420] Further, the display unit 2403 is further configured to display the invisible window, the transparency of the invisible window being greater than the threshold value, and the invisible window being used to initiate the drag event.

[0421] Further, the long press operation includes a press-down operation and a lift-up operation. The interception unit 2405 is configured to intercept the press-down event after the user performs the press-down operation by using the input device of the first terminal.

[0422] The sending unit 2401 is specifically configured to send the fourth operation parameter included in the press-down event to the second terminal.

[0423] The interception unit 2405 is further configured to intercept the lift-up event after the user performs the lift-up operation by using the input device of the first terminal after performing the press-down operation for a period of time.

[0424] The sending unit 2401 is further configured to send the fifth operation parameter included in the lift-up event to the second terminal.

[0425] The fourth operation parameter and the fifth operation parameter are respectively used for the second terminal to simulate the press-down event and the lift-up event, and further used to trigger the interface windowing of the application.

[0426] Embodiments of the present application further provide a cross-device object dragging device, which can be applied to the first terminal or the second terminal in the above-mentioned embodiments. The device can include a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement various functions or steps performed by the PC or the mobile phone or the tablet computer in the above-mentioned method embodiments when executing the instructions.

[0427] Embodiments of the present application further provide a terminal (which can be the first terminal or the second terminal in the above-mentioned embodiments), which can include a display screen, a memory and one or more processors. The display screen, the memory and the processor are coupled. The memory is used to store computer program codes, the computer program codes including computer instructions. When the processor executes the computer instructions, the terminal can perform various functions or steps performed by the PC or the mobile phone or the tablet computer in the above-mentioned method embodiments. Of course, the terminal includes but is not limited to the above-mentioned display screen, memory and one or more processors. For example, the structure of the terminal can refer to Figure 2AThe structure of the mobile phone shown.

[0428] The chip system can be applied to the terminal (such as the first terminal or the second terminal) in the foregoing embodiments. As shown in the figure, the chip system includes at least one processor 2501 and at least one interface circuit 2502. The processor 2501 can be the processor in the terminal. The processor 2501 and the interface circuit 2502 can be interconnected through a line. The processor 2501 can receive and execute computer instructions from the memory of the terminal through the interface circuit 2502. When the computer instructions are executed by the processor 2501, the terminal can perform the steps performed by the PC or the mobile phone or the tablet computer in the foregoing embodiments. Of course, the chip system can also include other discrete devices, and the embodiments of the present application do not specifically limit this. Figure 25

[0429] The embodiments of the present application further provide a computer readable storage medium for storing computer instructions run by the terminal (such as the PC or the mobile phone or the tablet computer).

[0430] The embodiments of the present application further provide a computer program product including computer instructions run by the terminal (such as the PC or the mobile phone or the tablet computer).

[0431] Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0432] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0433] The units described as separate components can or can not be physically separated, and the components shown as units can be one physical unit or multiple physical units, that is, can be located in one place or distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments. ​

[0434] In addition, each of the functional units in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0435] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solutions of the embodiments of the present application essentially, or the part that contributes to the prior art, or all or a part of the technical solutions can be embodied in the form of a software product. The software product is stored in a storage medium, including a number of instructions to make a device (which can be a single-chip microcomputer, a chip, etc.) or a processor execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and various other media that can store program codes.

[0436] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any change or replacement within the technical scope disclosed in the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A cross-device object-dragging method, characterized by, The method is applied to a second terminal, the second terminal is connected with a first terminal, the first terminal and the second terminal run different operating systems, and the method comprises the following steps: The second terminal creates a virtual input device; The second terminal displays a first cursor on an object displayed on the second terminal; the second terminal receives first operation parameters from the first terminal, the first operation parameters are operation parameters contained in a press event received by the first terminal after a user performs a press operation by using an input device of the first terminal, and the first operation parameters comprise one or more of a mouse key flag bit, coordinate information, scroll wheel information, and key information; The second terminal receives second operation parameters from the first terminal, the second operation parameters are operation parameters contained in a move event received by the first terminal after the user performs a move operation by using the input device of the first terminal, and the second operation parameters comprise one or more of a mouse key flag bit, coordinate information, scroll wheel information, and key information; The second terminal simulates the press event according to the first operation parameters and simulates the move event according to the second operation parameters by using the virtual input device, wherein the press event and the move event are used to indicate that a drag operation on the object is initiated; In response to the press event and the move event, the second terminal displays an animation of a shadow of the object moving with the first cursor on a display screen of the second terminal; The second terminal sends drag data to the first terminal.

2. The method of claim 1, wherein, The object is text or a file; the drag data comprises drag event content and a bitmap of the shadow; when the object is text, the drag event content comprises the text; and when the object is a file, the drag event content is a file path.

3. The method according to claim 1 or 2, characterized in that, The drag data is used by the first terminal to display an animation of the object moving with a second cursor on a display screen of the first terminal.

4. The method according to claim 1 or 2, characterized in that, The method further comprises the following steps: The second terminal creates a view control with a transparency greater than a threshold value, and the view control is used to initiate a drag event.

5. The method according to claim 1 or 2, characterized in that, After the step of displaying the animation of the object moving with the first cursor on the display screen of the second terminal in response to the press event and the move event, the method further comprises the following steps: The second terminal acquires and caches the drag data; The step of sending the drag data to the first terminal comprises the following steps: The second terminal sends the cached drag data to the first terminal.

6. The method of claim 5, wherein, Before the step of sending the drag data to the first terminal, the method further comprises the following step: The second terminal receives a request message from the first terminal, and the request message is used to request the drag data.

7. The method according to claim 1 or 2, characterized in that, Before the step of displaying the first cursor on the object displayed on the second terminal, the method further comprises the following steps: The second terminal receives first shuttle state information from the first terminal, and the first shuttle state information is used to indicate that a shuttle is started; In response to the first shuttle state information, the second terminal displays the first cursor on the second terminal.

8. The method of claim 7, wherein, After the second terminal displays the first cursor on the second terminal, the method further comprises: The second terminal receives an operation of moving the first cursor input by a user using an input device of the first terminal; In response to the operation of moving the first cursor, the second terminal displays an animation of the movement of the first cursor on the display screen of the second terminal until the first cursor moves onto the object.

9. The method of claim 8, wherein, The second terminal receives an operation of moving the first cursor input by a user using an input device of the first terminal, comprising: The second terminal receives a third operation parameter from the first terminal, the third operation parameter being an operation parameter contained in a movement event received by the first terminal after a user performs a movement operation using an input device of the first terminal; The second terminal simulates the movement event according to the third operation parameter, the movement event being used to indicate the movement of the first cursor.

10. The method of claim 1 or 2, wherein, After the second terminal determines that the object is dragged out of the edge of the display screen of the second terminal, the method further comprises: The second terminal sends second shuttle state information to the first terminal, the second shuttle state information being used to indicate the end of the shuttle.

11. The method according to any one of claims 1-10, characterized in that, The second terminal creates a virtual input device, comprising: The second terminal creates a virtual input device after the connection with the first terminal is successfully established; or, The second terminal receives a notification message from the first terminal, and in response to the notification message, the second terminal creates the virtual input device.

12. A cross-device object-dragging method, comprising: Applied to a first terminal, the first terminal is connected with a second terminal, the operating system running on the first terminal is different from the operating system running on the second terminal, and the method comprises: After a user performs the pressing operation using an input device of the first terminal, the first terminal intercepts a pressing event corresponding to the pressing operation; The first terminal sends a first operation parameter included in the pressing event to the second terminal, the first operation parameter comprising one or more of a mouse key flag bit, coordinate information, scroll wheel information, and key information; After a user performs the movement operation using an input device of the first terminal, the first terminal intercepts a movement event corresponding to the movement operation; The first terminal sends a second operation parameter included in the movement event to the second terminal, the second operation parameter comprising one or more of a mouse key flag bit, coordinate information, scroll wheel information, and key information; The first operation parameter and the second operation parameter are respectively used by the second terminal to simulate the pressing event and the movement event by using a virtual input device, and are further used to indicate the initiation of the drag operation on the object displayed by the second terminal, the virtual input device being created by the second terminal; The first terminal receives drag data from the second terminal, the drag data being sent by the second terminal to the first terminal; The first terminal displays an animation of the shadow of the object moving with a second cursor on the display screen of the first terminal according to the drag data, the pressing event, and the movement event.

13. The method of claim 12, wherein, The object is text or a file; the drag data includes drag event content and a bitmap of the shadow; when the object is text, the drag event content includes the text, and when the object is a file, the drag event content is a file path.

14. The method according to claim 12 or 13, characterized in that, Before the first terminal sends the first operation parameter included in the press event and the second operation parameter included in the move event to the second terminal, the method further includes: The first terminal determines that the second cursor slides out of the edge of the display screen of the first terminal.

15. The method of claim 14, wherein, Before the first terminal determines that the second cursor slides out of the edge of the display screen of the first terminal, the method further includes: The first terminal receives an operation of moving the second cursor input by a user using an input device of the first terminal; In response to the operation of moving the second cursor, the first terminal displays an animation of the movement of the second cursor on the display screen of the first terminal.

16. The method of claim 14, wherein, After the first terminal determines that the second cursor slides out of the edge of the display screen of the first terminal, the method further includes: The first terminal sends first shuttle state information to the second terminal, and the first shuttle state information is used to indicate the start of shuttle.

17. The method of claim 14, wherein, After the first terminal determines that the second cursor slides out of the edge of the display screen of the first terminal, the method further includes: The first terminal sends data of an operation of moving the first cursor input by a user using an input device of the first terminal to the second terminal, which is used to display an animation of the movement of the first cursor on the display screen of the second terminal.

18. The method of claim 17, wherein, The first terminal sends data of an operation of moving the first cursor input by a user using an input device of the first terminal to the second terminal, which includes: After a user performs a moving operation using an input device of the first terminal, the first terminal intercepts a move event; The first terminal sends a third operation parameter included in the move event to the second terminal, and the third operation parameter is used to simulate the move event by the second terminal, and is further used to indicate the movement of the first cursor.

19. The method of claim 12 or 13, wherein, Before the first terminal displays an animation of the movement of the shadow of the object along with the second cursor on the display screen of the first terminal according to the drag data, the press event and the move event, the method further includes: The first terminal receives second shuttle state information from the second terminal, and the second shuttle state information is used to indicate the end of shuttle.

20. The method of claim 19, wherein, Before the first terminal receives the drag data from the second terminal, the method further includes: The first terminal sends a request message to the second terminal, and the request message is used to request the drag data.

21. The method of claim 19, wherein, After the first terminal receives the second shuttle state information from the second terminal, the method further includes: The first terminal cancels the interception of the input event; The first terminal displays an animation of the movement of the object along with the second cursor on the display screen of the first terminal according to the drag data, the press event and the move event, which includes: The first terminal receives a move event after a user performs a move operation on an input device of the first terminal; The first terminal generates a press event; The first terminal displays an animation of the object moving with the second cursor on a display screen of the first terminal according to the move event, the press event and the drag data.

22. The method of claim 14, wherein, The method further comprises: The first terminal displays an invisible window, the invisible window having a transparency greater than a threshold value, the invisible window being used to initiate a drag event.

23. A cross-device object dragging device, characterized in that, Comprise: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the method of any one of claims 1-11, or implement the method of any one of claims 12-22 when executing the instructions.

24. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by a processor, implement the method of any one of claims 1-11, or implement the method of any one of claims 12-22.

25. A cross-device object-dragging system, comprising: Comprise: a first terminal and a second terminal; the first terminal is connected with the second terminal, and the first terminal and the second terminal run different operating systems; The second terminal is configured to create a virtual input device; display a first cursor on an object displayed on the second terminal; receive a first operation parameter from the first terminal, the first operation parameter being an operation parameter contained in a press event received by the first terminal after a user performs a press operation on an input device of the first terminal, the first operation parameter comprising one or more of a mouse key flag bit, coordinate information, scroll wheel information and key information; receive a second operation parameter from the first terminal, the second operation parameter being an operation parameter contained in a move event received by the first terminal after a user performs a move operation on the input device of the first terminal, the second operation parameter comprising one or more of a mouse key flag bit, coordinate information, scroll wheel information and key information; simulate the press event according to the first operation parameter and simulate the move event according to the second operation parameter using the virtual input device, the press event and the move event being used to indicate initiation of a drag on the object; in response to the press event and the move event, display an animation of a shadow of the object moving with the first cursor on a display screen of the second terminal; send drag data to the first terminal; The first terminal is configured to intercept the press event and the move event; receive drag data from the second terminal; display an animation of the shadow of the object moving with a second cursor on a display screen of the first terminal according to the drag data, the press event and the move event.

26. The system of claim 25, wherein, The object is text or a file; the drag data comprises drag event content and a bitmap of the shadow; when the object is text, the drag event content comprises the text; when the object is a file, the drag event content is a file path.

27. The system of claim 25 or 26, wherein, The second terminal is further configured to obtain and cache the drag data; The sending the drag data to the first terminal comprises: sending the buffered drag data to the first terminal.

28. The system of claim 25 or 26, wherein, The first terminal is further configured to determine that the second cursor slides out of the edge of the display screen of the first terminal.

29. The system of claim 25 or 26, wherein, The first terminal is further configured to, after determining that the second cursor slides out of the edge of the display screen of the first terminal, send, to the second terminal, data of a drag operation input by a user using an input device of the first terminal.

30. The system of claim 25 or 26, wherein, The first terminal is further configured to, after determining that the second cursor slides out of the edge of the display screen of the first terminal, send, to the second terminal, first shuttle state information, the first shuttle state information being used to indicate a start of the shuttle.

31. The system of claim 30, wherein, The second terminal is further configured to display the first cursor on the second terminal, receive an operation of moving the first cursor input by a user using an input device of the first terminal, and in response to the operation of moving the first cursor, display an animation of movement of the first cursor on the display screen of the second terminal until the first cursor moves onto the object.

32. The system of claim 31, wherein, The receiving the operation of moving the first cursor input by the user using the input device of the first terminal comprises: receiving third operation parameters from the first terminal, the third operation parameters being operation parameters contained in a movement event received by the first terminal after the user performs the movement operation using the input device of the first terminal; simulating the movement event according to the third operation parameters, the movement event being used to indicate the movement of the first cursor.

33. The system of claim 25 or 26, wherein, The second terminal is further configured to, after determining that the object is dragged out of the edge of the display screen of the second terminal, send, to the first terminal, second shuttle state information, the second shuttle state information being used to indicate an end of the shuttle.

34. The system of any one of claims 25-33, wherein, The creating the virtual input device comprises: creating the virtual input device after the connection with the first terminal is successfully established; or receiving a notification message from the first terminal, and in response to the notification message, creating the virtual input device.

Images