A cross-device object dragging method and device

Abstract

The present application provides a cross-device object dragging method and device, which relates to the field of electronic devices. The efficiency of using multiple terminals in a coordinated manner is improved, and the interaction between terminals when multiple terminals are used in a coordinated manner is increased. The specific scheme is: the second terminal displays a first cursor on the object displayed on the second terminal; the second terminal receives a drag operation input by a user using an input device of the first terminal, and the drag operation is used to initiate dragging of the object; in response to the drag operation, the second terminal displays an animation of the object moving with the first cursor on the display screen of the second terminal; when the second terminal determines that the object is dragged out of the edge of the display screen of the second terminal, it sends drag data to the first terminal for the first terminal to continue dragging.

Description

Technical Field

[0001] The present application relates to the field of electronic devices, and in particular to a method and device for dragging objects across devices. Background Art

[0002] With the development of electronic technology and mobile Internet, a user can have more terminals such as mobile phones, tablets, personal computers (PCs), smart home devices (such as TVs) at the same time. In general, the use of each terminal is relatively independent. In scenarios where there is a need for collaborative use of multiple terminals, such as collaborative office, users will connect multiple terminals together for use. For example, a user has a PC and a mobile phone. The user can connect the PC and the mobile phone wirelessly or wiredly to use them together to achieve collaborative office between the PC and the mobile phone. Summary of the invention

[0003] An embodiment of the present application provides a cross-device object dragging method. In a scenario where multiple terminals (such as a PC and a mobile phone) are used in collaboration, keyboard and mouse sharing technology is used to implement the dragging of objects such as application icons, application interfaces, files, texts, or folders in the mobile phone from the mobile phone to the PC, thereby improving the efficiency of the collaborative use of the PC and the mobile phone and increasing the interaction between the terminals when multiple terminals are used in collaboration.

[0004] In order to achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0005] In a first aspect of the present application, a method for dragging an object across devices is provided. The method can be applied to a second terminal, which is connected to a first terminal. The method may include: the second terminal displays a first cursor on an object displayed on the second terminal; the second terminal receives a drag operation input by a user using an input device of the first terminal, wherein the drag operation is used to initiate dragging of the object; in response to the drag operation, the second terminal displays an animation of the object moving with the first cursor on a display screen of the second terminal; and when the second terminal determines that the object is dragged out of an edge of the display screen of the second terminal, the second terminal sends drag data to the first terminal.

[0006] As an example, the input device of the first terminal may be a mouse, a touch pad, etc. The first cursor may be a cursor displayed on a display screen of the second terminal.

[0007] By adopting the above technical solution, in the scenario where multiple terminals are used in collaboration, the user can use input devices such as a mouse to drag and drop objects on one terminal to follow the cursor between multiple terminals participating in the collaborative use. For example, the user can use the input device of one terminal to drag objects on other terminals from the other terminals to the terminal, which improves the efficiency of the collaborative use of multiple terminals, increases the interaction between the terminals when multiple terminals are used in collaboration, and improves the user experience.

[0008] In a possible implementation, the drag data can be used by the first terminal to display an animation of an object moving with a second cursor on a display screen of the first terminal. The second cursor can be a cursor displayed on the display screen of the first terminal. By sending the drag data to other terminals, the other terminals can continue to display an animation of an object moving with a cursor on the other terminals according to the operation input by the user, thereby realizing the continuation of dragging.

[0009] In another possible implementation, the object may be an application icon; or, the object may be a window including an application interface; when the object is an application icon, the drag data may include: the application icon; when the object is a window, the drag data may include: the application interface.

[0010] In another possible implementation, after sending the drag data to the first terminal, the method may further include: the second terminal receives an indication message from the first terminal, the indication message is sent by the first terminal after the user uses the input device of the first terminal to input the drag release operation, and is used to indicate that the dragging of the object is finished; the second terminal sends data to the first terminal, and the data is used for the first terminal to display the interface of the application on the display screen of the first terminal. After the dragging of the object is finished, the corresponding data is sent to the first terminal so that the first terminal can display the interface of the application corresponding to the dragged object on the display screen of the first terminal.

[0011] In another possible implementation, when the object is an icon of an application, after the second terminal receives a drag operation input by the user using an input device of the first terminal, the method may further include: the second terminal creates a drag object according to the icon of the application, and the drag object may include: an icon identical to the icon of the application; in response to the drag operation, the second terminal displays an animation of the object moving with the first cursor on the display screen of the second terminal, including: in response to the drag operation, the second terminal displays an animation of the drag object moving with the first cursor. The created icon identical to the icon of the dragged application can be used for dragging.

[0012] In another possible implementation, when the object is a window, before the second terminal displays the first cursor on the object displayed on the second terminal, the method may also include: the second terminal displays the interface of the application; when the first cursor is displayed on the status bar of the second terminal, the second terminal receives a long press operation input by the user using an input device of the first terminal to trigger the windowing of the interface of the application; in response to the long press operation, the second terminal displays a window; the window specifically includes a screenshot of the interface of the application, and the above-mentioned drag data specifically includes the screenshot of the interface of the application.

[0013] In another possible implementation, the second terminal displays an animation of the object moving with the first cursor on the display screen of the second terminal, which may include: the second terminal displays an animation of the shadow of the object moving with the first cursor on the display screen of the second terminal.

[0014] In another possible implementation, the above-mentioned object can be text, a file or a folder; the drag data includes the drag event content and a bitmap of the shadow; wherein, when the object is text, the drag event content includes text, and when the object is a file or a folder, the drag event content is the file path.

[0015] In another possible implementation, after the second terminal displays an animation of the object moving along with the first cursor on a display screen of the second terminal in response to the drag operation, the method may further include: the second terminal acquiring and caching the drag data.

[0016] In another possible implementation, before sending the drag data to the first terminal, the method further includes: the second terminal receiving a request message from the first terminal, the request message being used to request the drag data; sending the drag data to the first terminal may include: the second terminal sending cached drag data to the first terminal.

[0017] In another possible implementation, before the second terminal displays the first cursor on the object displayed on the second terminal, the method may further include: the second terminal receives first shuttle status information from the first terminal, the first shuttle status information is used to indicate the start of shuttle; in response to the first shuttle status information, the second terminal displays the first cursor on the second terminal. After receiving the shuttle status information indicating the start of shuttle, the user can use the input device of the first terminal to control the second terminal.

[0018] In another possible implementation, after the second terminal displays the first cursor on the second terminal, the method may further include: the second terminal receives an operation of moving the first cursor input by the user using an input device of the first terminal; in response to the operation of moving the first cursor, the second terminal displays an animation of the movement of the first cursor on the display screen of the second terminal until the first cursor moves to the object. After the cursor shuttle starts, the user can use the input device of the first terminal to control the second terminal.

[0019] In another possible implementation, the drag operation may include a press operation and a move operation; the second terminal receives the drag operation input by the user using the input device of the first terminal, including: the second terminal receives a first operation parameter from the first terminal, the first operation parameter is an operation parameter included in a press event received by the first terminal after the user uses the input device of the first terminal to perform a press operation; the second terminal receives a second operation parameter from the first terminal, the second operation parameter is an operation parameter included in a move event received by the first terminal after the user uses the input device of the first terminal to perform a move operation; the second terminal simulates a press event according to the first operation parameter, and simulates a move event according to the second operation parameter, the press event and the move event are used to indicate the initiation of dragging the object; in response to the drag operation, the second terminal displays an animation of the object moving with the first cursor on a display screen of the second terminal, including: in response to the press event and the move event, the second terminal displays an animation of the object moving with the first cursor.

[0020] In another possible implementation, the second terminal receives an operation of moving the first cursor input by the user using the input device of the first terminal, which may include: the second terminal receives a third operation parameter from the first terminal, the third operation parameter is an operation parameter included in a movement event received by the first terminal after the user uses the input device of the first terminal to perform the movement operation; the second terminal simulates a movement event according to the third operation parameter, and the movement event is used to indicate the movement of the first cursor.

[0021] In another possible implementation, after the second terminal determines that the object is dragged out of the edge of the display screen of the second terminal, the method may further include: the second terminal sends second shuttle state information to the first terminal, where the second shuttle state information is used to indicate the end of the shuttle. By sending the shuttle state information indicating the end of the shuttle to the first terminal, the user can use the input device of the first terminal to control the first terminal.

[0022] In another possible implementation, the long press operation may include a press operation and a lift operation; the second terminal receives the long press operation input by the user using the input device of the first terminal, which may include: the second terminal receives a fourth operation parameter from the first terminal, the fourth operation parameter is when the first cursor is displayed on the status bar, and the user uses the input device of the first terminal to perform a press operation. The operation parameter contained in the press event received by the first terminal after the user uses the input device of the first terminal to perform a lift operation after performing the press operation for a period of time; the second terminal simulates a press event according to the fourth operation parameter, and simulates a lift event according to the fifth operation parameter, and the press event and the lift event are used to trigger the interface windowing of the application; in response to the long press operation, the second terminal displays a window, including: in response to the press event and the lift event, the second terminal displays a window.

[0023] In another possible implementation, the method may further include: after the second terminal successfully establishes a connection with the first terminal, the second terminal creates a virtual input device; or, the second terminal receives a notification message from the first terminal, the notification message is used to indicate that the keyboard and mouse sharing mode of the first terminal has been turned on, and in response to the notification message, the second terminal creates a virtual input device; wherein the virtual input device is used for the second terminal to simulate input events according to the operation parameters. With the help of the keyboard and mouse sharing technology, it is possible to use the input device of one terminal to drag objects between multiple terminals.

[0024] According to a second aspect of the present application, a method for dragging an object across devices is provided. The method can be applied to a first terminal, which is connected to a second terminal. The method may include: the first terminal sends to the second terminal data of a drag operation input by a user using an input device of the first terminal, wherein the drag operation is used to initiate dragging of an object displayed on the second terminal; the first terminal receives drag data from the second terminal, wherein the drag data is sent to the first terminal after the second terminal determines that the object is dragged out of the edge of the display screen of the second terminal during a process in which the object moves with a first cursor on the display screen of the second terminal.

[0025] As an example, the input device of the first terminal may be a mouse, a touch pad, etc. The first cursor may be a cursor displayed on a display screen of the second terminal.

[0026] By adopting the above technical solution, in the scenario where multiple terminals are used in collaboration, the user can use input devices such as a mouse to drag and drop objects on one terminal to follow the cursor between multiple terminals participating in the collaborative use. For example, the user can use the input device of one terminal to drag objects on other terminals from the other terminals to the terminal, which improves the efficiency of the collaborative use of multiple terminals, increases the interaction between the terminals when multiple terminals are used in collaboration, and improves the user experience.

[0027] In a possible implementation, after the first terminal receives the drag data from the second terminal, the method may further include: the first terminal displays an animation of the object moving with the second cursor on the display screen of the first terminal according to the drag data and the drag operation. The second cursor may be a cursor displayed on the display screen of the first terminal. According to the received drag data and the operation input by the user, the animation of the object moving with the cursor may continue to be displayed on the first terminal, thereby realizing the continuation of the drag.

[0028] In another possible implementation, the object may be an application icon; or, the object may be a window including an application interface; when the object is an application icon, the drag data may include: the application icon; when the object is a window, the drag data may include: the application interface.

[0029] In another possible implementation, after the first terminal receives the drag data from the second terminal, the method further includes: the first terminal receives a drag release operation input by the user using an input device of the first terminal; in response to the drag release operation, the first terminal displays the interface of the application on the display screen of the first terminal. After the dragging of the object ends, the first terminal may display the interface of the application corresponding to the dragged object on the display screen of the first terminal.

[0030] In another possible implementation, the first terminal displays the interface of the application on the display screen of the first terminal, which may include: the first terminal sends an indication message to the second terminal, where the indication message can be used to indicate the end of dragging the object; the first terminal receives data from the second terminal; and the first terminal displays the interface of the application on the display screen of the first terminal based on the data.

[0031] In another possible implementation, when the object is a window, before the first terminal sends data of a drag operation input by a user using an input device of the first terminal to the second terminal, the method may further include: the first terminal sends data of a long press operation input by a user using an input device of the first terminal to the second terminal, where the long press operation is input by the user when a first cursor is displayed on a status bar of the second terminal, and is used to trigger the windowing of the interface of an application displayed on the second terminal; the window specifically includes a screenshot of the interface of the application, and the drag data specifically includes a screenshot of the interface of the application.

[0032] In another possible implementation, the first terminal displays an animation of the object moving with the second cursor on the display screen of the first terminal according to the drag data and the drag operation, which may include: the first terminal displays an animation of the shadow of the object moving with the second cursor on the display screen of the first terminal according to the drag data and the drag operation.

[0033] In another possible implementation, the above-mentioned object can be text, a file or a folder; the drag data includes the drag event content and a bitmap of the shadow; wherein, when the object is text, the drag event content includes text, and when the object is a file or a folder, the drag event content is the file path.

[0034] In another possible implementation, before the first terminal sends the drag operation data input by the user using the input device of the first terminal to the second terminal, the method may further include: the first terminal determines that the second cursor slides out of the edge of the display screen of the first terminal. The user triggers the cursor shuttle by sliding the cursor on the display screen of the first terminal out of the edge of the display screen of the first terminal.

[0035] In another possible implementation, before the first terminal determines that the second cursor slides out of the edge of the display screen of the first terminal, the method may also include: the first terminal receives an operation of moving the second cursor input by the user using an input device of the first terminal; in response to the operation of moving the second cursor, the first terminal displays an animation of the movement of the second cursor on the display screen of the first terminal.

[0036] In another possible implementation, after the first terminal determines that the second cursor slides out of the edge of the display screen of the first terminal, the method may further include: the first terminal sends first shuttle state information to the second terminal, the first shuttle state information is used to indicate the start of the shuttle. After the cursor shuttles, the shuttle state information indicating the start of the shuttle is sent to other terminals, so that the other terminals are prepared for the continued dragging, such as displaying the cursor.

[0037] In another possible implementation, after the first terminal determines that the second cursor slides out of the edge of the display screen of the first terminal, the method may further include: the first terminal sends data of the operation of moving the first cursor input by the user using the input device of the first terminal to the second terminal, so that the second terminal displays the animation of the movement of the first cursor on the display screen of the second terminal. After the cursor shuttles, the first terminal sends the operation input by the user using the first terminal to other terminals, so as to realize the control of the other terminals by the input device of the first terminal.

[0038] In another possible implementation, the above-mentioned drag operation may include a pressing operation and a moving operation; the first terminal sends data of the drag operation input by the user using the input device of the first terminal to the second terminal, including: after the user uses the input device of the first terminal to perform a pressing operation, the first terminal intercepts the pressing event; the first terminal sends a first operation parameter included in the pressing event to the second terminal; after the user uses the input device of the first terminal to perform a moving operation, the first terminal intercepts the moving event; the first terminal sends a second operation parameter included in the moving event to the second terminal; wherein the first operation parameter and the second operation parameter are respectively used by the second terminal to simulate a pressing event and a moving event, and then used to indicate the initiation of dragging the object.

[0039] In another possible implementation, the first terminal sends data of an operation of moving a first cursor input by a user using an input device of the first terminal to a second terminal, including: after the user performs a moving operation using the input device of the first terminal, the first terminal intercepts the moving event; the first terminal sends a third operation parameter included in the moving event to the second terminal, and the third operation parameter is used by the second terminal to simulate the moving event, and then to indicate the movement of the first cursor.

[0040] In another possible implementation, before the first terminal displays an animation of the object moving with the second cursor on the display screen of the first terminal according to the drag data and the drag operation, the method may further include: the first terminal receives second shuttle status information from the second terminal, the second shuttle status information being used to indicate the end of the shuttle.

[0041] In another possible implementation manner, before the first terminal receives the drag data from the second terminal, the method may further include: the first terminal sends a request message to the second terminal, where the request message is used to request the drag data.

[0042] In another possible implementation, after the first terminal receives the second shuttle state information from the second terminal, the method may further include: the first terminal cancels the interception of the input event; the first terminal displays an animation of the object moving with the second cursor on the display screen of the first terminal according to the drag operation and the drag data, including: after the user uses the input device of the first terminal to perform a move operation, the first terminal receives a move event; the first terminal generates a press event; the first terminal displays an animation of the object moving with the second cursor on the display screen of the first terminal according to the move event, the press event and the drag data.

[0043] In another possible implementation, after the first terminal determines that the second cursor slides out of the edge of the display screen of the first terminal, the method may further include: the first terminal displays an invisible window, the transparency of the invisible window is greater than a threshold, and the invisible window is used to initiate a drag event.

[0044] In another possible implementation, the long press operation includes a press operation and a lift operation; the first terminal sends data of the long press operation input by the user using the input device of the first terminal to the second terminal, including: after the user uses the input device of the first terminal to perform a press operation, the first terminal intercepts the press event, and the first terminal sends a fourth operation parameter included in the press event to the second terminal; after the user performs the press operation for a period of time and then uses the input device of the first terminal to perform a lift operation, the first terminal intercepts the lift event, and the first terminal sends a fifth operation parameter included in the lift event to the second terminal; wherein the fourth operation parameter and the fifth operation parameter are respectively used by the second terminal to simulate a press event and a lift event, and then used to trigger the windowing of the application interface.

[0045] According to a third aspect of the present application, a cross-device object dragging apparatus is provided, which can be applied to a second terminal, where the second terminal is connected to a first terminal, and the apparatus may include: a display unit, configured to display a first cursor on an object displayed on the second terminal; a receiving unit, configured to receive a drag operation input by a user using an input device of the first terminal, where the drag operation is used to initiate dragging of the object; the display unit, configured to display an animation of the object moving with the first cursor on a display screen of the second terminal in response to the drag operation; and a sending unit, configured to send drag data to the first terminal when it is determined that the object is dragged out of the edge of the display screen of the second terminal.

[0046] In a possible implementation, the drag data may be used by the first terminal to display an animation of the object moving along with the second cursor on a display screen of the first terminal.

[0047] In another possible implementation, the object may be an application icon; or, the object may be a window including an application interface; when the object is an application icon, the drag data includes: the application icon; when the object is a window, the drag data includes: the application interface.

[0048] In another possible implementation, the receiving unit is further used to receive an indication message from the first terminal, where the indication message is sent by the first terminal after the user uses an input device of the first terminal to input a drag-and-release operation, and is used to indicate the end of dragging the object; the sending unit is further used to send data to the first terminal, where the data is used by the first terminal to display the interface of the application on the display screen of the first terminal.

[0049] In another possible implementation, when the object is an application icon, the device also includes: a creation unit; a creation unit, used to create a drag object based on the application icon, and the drag object includes: an icon that is the same as the application icon; a display unit, specifically used to display an animation of the drag object moving with the first cursor in response to the drag operation.

[0050] In another possible implementation, when the object is a window, the display unit is also used to display the interface of the application; the receiving unit is also used to receive a long press operation input by the user using an input device of the first terminal when the first cursor is displayed on the status bar of the second terminal, so as to trigger the windowing of the interface of the application; the display unit is also used to display the window in response to the long press operation; the window specifically includes a screenshot of the interface of the application, and the dragged data specifically includes a screenshot of the interface of the application.

[0051] In another possible implementation manner, the display unit is specifically configured to display an animation of a shadow of the object moving along with the movement of the first cursor on a display screen of the second terminal.

[0052] In another possible implementation, the object may be text, a file or a folder; the drag data includes the drag event content and a shadow bitmap; wherein, when the object is text, the drag event content includes text, and when the object is a file or a folder, the drag event content is the file path.

[0053] In another possible implementation, the device may further include: an acquisition unit, configured to acquire and cache the drag data.

[0054] In another possible implementation, the receiving unit is further configured to receive a request message from the first terminal, where the request message is used to request the drag data; and the sending unit is specifically configured to send the cached drag data to the first terminal.

[0055] In another possible implementation, the receiving unit is further configured to receive first shuttle status information from the first terminal, the first shuttle status information being used to indicate the start of shuttling; and the display unit is further configured to display a first cursor on the second terminal in response to the first shuttle status information.

[0056] In another possible implementation, the receiving unit is further used to receive an operation of moving the first cursor input by the user using an input device of the first terminal; the display unit is further used to display an animation of the movement of the first cursor on a display screen of the second terminal in response to the operation of moving the first cursor until the first cursor moves onto the object.

[0057] In another possible implementation, the drag operation includes a press operation and a move operation; a receiving unit is specifically used to receive a first operation parameter from the first terminal, the first operation parameter is an operation parameter included in a press event received by the first terminal after the user uses an input device of the first terminal to perform a press operation; a second operation parameter is received from the first terminal, the second operation parameter is an operation parameter included in a move event received by the first terminal after the user uses the input device of the first terminal to perform a move operation; the device may also include: a simulation unit, used to simulate a press event according to the first operation parameter, and simulate a move event according to the second operation parameter, the press event and the move event are used to indicate the initiation of dragging an object; a display unit, specifically used to display an animation of the object moving with the first cursor in response to the press event and the move event.

[0058] In another possible implementation, the receiving unit is specifically used to receive a third operation parameter from the first terminal, where the third operation parameter is an operation parameter included in a mobile event received by the first terminal after the user uses an input device of the first terminal to perform a mobile operation; the device may also include: a simulation unit, used to simulate a mobile event according to the third operation parameter, and the mobile event is used to indicate the movement of the first cursor.

[0059] In another possible implementation, the sending unit is further configured to send second shuttling status information to the first terminal, where the second shuttling status information is used to indicate the end of the shuttling.

[0060] In another possible implementation, the long press operation includes a press operation and a lift operation; a receiving unit is specifically used to receive a fourth operation parameter from the first terminal, the fourth operation parameter is the operation parameter contained in the press event received by the first terminal after the user uses the input device of the first terminal to perform a press operation when the first cursor is displayed on the status bar; a fifth operation parameter is received from the first terminal, the fifth operation parameter is the operation parameter contained in the lift event received by the first terminal after the user uses the input device of the first terminal to perform a lift operation after performing the press operation for a period of time; the device may also include: a simulation unit, used to simulate a press event according to the fourth operation parameter, and simulate a lift event according to the fifth operation parameter, the press event and the lift event are used to trigger the windowing of the application interface; a display unit, specifically used to display a window in response to the press event and the lift event.

[0061] In another possible implementation, the creation unit is further used to create a virtual input device after the connection with the first terminal is successfully established; or, the receiving unit is further used to receive a notification message from the first terminal, the notification message is used to indicate that the keyboard and mouse sharing mode of the first terminal has been turned on, and the creation unit is further used to create a virtual input device in response to the notification message; wherein the virtual input device is used by the second terminal to simulate input events according to operating parameters.

[0062] According to a fourth aspect of the present application, a cross-device object dragging apparatus is provided, which is applied to a first terminal, and the first terminal is connected to a second terminal. The apparatus may include: a sending unit, which is used to send to the second terminal data of a drag operation input by a user using an input device of the first terminal, and the drag operation is used to initiate dragging of an object displayed on the second terminal; and a receiving unit, which is used to receive drag data from the second terminal, and the drag data is sent to the first terminal after the second terminal determines that the object is dragged out of the edge of the display screen of the second terminal during the process of the object moving with a first cursor on the display screen of the second terminal.

[0063] In a possible implementation, the device may further include: a display unit, configured to display an animation of the object moving along with the second cursor on a display screen of the first terminal according to the drag data and the drag operation.

[0064] In another possible implementation, the object may be an application icon; or, the object may be a window including an application interface; when the object is an application icon, the drag data includes: the application icon; when the object is a window, the drag data includes: the application interface.

[0065] In another possible implementation, the receiving unit is further configured to receive a drag-and-release operation input by the user using an input device of the first terminal; and the display unit is further configured to display an interface of the application on a display screen of the first terminal in response to the drag-and-release operation.

[0066] In another possible implementation, the sending unit is further used to send an indication message to the second terminal, where the indication message is used to indicate the end of dragging the object; the receiving unit is further used to receive data from the second terminal; and the display unit is specifically used to display the interface of the application on the display screen of the first terminal according to the data.

[0067] In another possible implementation, when the object is a window, the sending unit is further used to send data of a long press operation input by the user using an input device of the first terminal to the second terminal, where the long press operation is input by the user when the first cursor is displayed on the status bar of the second terminal, and is used to trigger the windowing of the interface of the application displayed on the second terminal; the window specifically includes a screenshot of the interface of the application, and the drag data specifically includes a screenshot of the interface of the application.

[0068] In another possible implementation, the display unit is specifically configured to display an animation of a shadow of the object moving along with the second cursor on a display screen of the first terminal according to the drag data and the drag operation.

[0069] In another possible implementation, the object may be text, a file or a folder; the drag data includes the drag event content and a shadow bitmap; wherein, when the object is text, the drag event content includes text, and when the object is a file or a folder, the drag event content is the file path.

[0070] In another possible implementation manner, the device may further include: a determining unit, configured to determine that the second cursor slides out of an edge of a display screen of the first terminal.

[0071] In another possible implementation, the receiving unit is further used to receive an operation of moving the second cursor input by the user using an input device of the first terminal; the display unit is further used to display an animation of the movement of the second cursor on the display screen of the first terminal in response to the operation of moving the second cursor.

[0072] In another possible implementation manner, the sending unit is further configured to send first shuttling state information to the second terminal, where the first shuttling state information is used to indicate the start of shuttling.

[0073] In another possible implementation, the sending unit is further configured to send data of an operation of moving the first cursor input by the user using an input device of the first terminal to the second terminal, so that the second terminal displays an animation of the movement of the first cursor on a display screen of the second terminal.

[0074] In another possible implementation, the drag operation includes a pressing operation and a moving operation; the device may also include: an intercepting unit; the intercepting unit is used to intercept a pressing event after the user uses the input device of the first terminal to perform a pressing operation; a sending unit is specifically used to send a first operation parameter included in the pressing event to the second terminal; the intercepting unit is also used to intercept a moving event after the user uses the input device of the first terminal to perform a moving operation; the sending unit is specifically used to send a second operation parameter included in the moving event to the second terminal; wherein the first operation parameter and the second operation parameter are respectively used by the second terminal to simulate a pressing event and a moving event, and then used to indicate the initiation of dragging the object.

[0075] In another possible implementation, the interception unit is used to intercept the move event after the user uses the input device of the first terminal to perform a move operation; the sending unit is specifically used to send a third operation parameter included in the move event to the second terminal, and the third operation parameter is used for the second terminal to simulate the move event, and then to instruct the movement of the first cursor.

[0076] In another possible implementation, the receiving unit is further configured to receive second shuttling status information from the second terminal, where the second shuttling status information is used to indicate the end of shuttling.

[0077] In another possible implementation, the sending unit is further configured to send a request message to the second terminal, where the request message is used to request to drag the data.

[0078] In another possible implementation, the device may also include: an interception unit and an input unit; the interception unit is used to cancel the interception of the input event; the input unit is used to receive the move event after the user uses the input device of the first terminal to perform the move operation; the input unit is also used to generate a press event; the display unit is specifically used to display an animation of the object moving with the second cursor on the display screen of the first terminal according to the move event, the press event and the drag data.

[0079] In another possible implementation, the display unit is further used to display an invisible window, the transparency of the invisible window is greater than a threshold, and the invisible window is used to initiate a drag event.

[0080] In another possible implementation, the long press operation includes a press operation and a lift operation; the interception unit is used to intercept the press event after the user uses the input device of the first terminal to perform the press operation; the sending unit is specifically used to send the fourth operation parameter included in the press event to the second terminal; the interception unit is also used to intercept the lift event after the user uses the input device of the first terminal to perform the lift operation after performing the press operation for a period of time; the sending unit is also used to send the fifth operation parameter included in the lift event to the second terminal; wherein the fourth operation parameter and the fifth operation parameter are respectively used by the second terminal to simulate the press event and the lift event, and then used to trigger the windowing of the application interface.

[0081] In a fifth aspect of the present application, a cross-device object dragging apparatus is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement a method as described in the first aspect or any one of the possible implementations of the first aspect, or to implement a method as described in the second aspect or any one of the possible implementations of the second aspect, when executing the instructions.

[0082] In a sixth aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, a method as described in the first aspect or any one of the possible implementations of the first aspect is implemented when the instructions are executed, or a method as described in the second aspect or any one of the possible implementations of the second aspect is implemented.

[0083] In the seventh aspect of the present application, a terminal is provided, which includes a display screen, one or more processors and a memory; the display screen, the processor and the memory are coupled; the memory is used to store computer program code, the computer program code includes computer instructions, and when the computer instructions are executed by the terminal, the terminal executes a method as described in any one of the first aspect or possible implementations of the first aspect, or the terminal executes a method as described in any one of the second aspect or possible implementations of the second aspect.

[0084] In an eighth aspect of the present application, a computer program product is provided, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code, wherein when the computer-readable code is executed in a terminal, a processor in the terminal executes a method according to the first aspect or any one of the possible implementations of the first aspect, or executes a method according to the second aspect or any one of the possible implementations of the second aspect.

[0085] In a ninth aspect of the present application, a cross-device object dragging system is provided, which may include a first terminal and a second terminal, wherein the first terminal is connected to the second terminal.

[0086] The second terminal displays a first cursor on an object displayed on the second terminal; the first terminal sends data of a drag operation input by a user using an input device of the first terminal to the second terminal; the second terminal receives a drag operation input by a user using an input device of the first terminal, the drag operation being used to initiate dragging of the object; in response to the drag operation, the second terminal displays an animation of the object moving along with the first cursor on a display screen of the second terminal; when determining that the object is dragged out of an edge of a display screen of the second terminal, the second terminal sends drag data to the first terminal. The first terminal receives the drag data from the second terminal.

[0087] In a possible implementation manner, the first terminal displays an animation of the object moving along with the second cursor on a display screen of the first terminal according to the drag data and the drag operation.

[0088] In another possible implementation, after determining that the second cursor slides out of the edge of the display screen of the first terminal, the first terminal sends data of the drag operation input by the user using the input device of the first terminal to the second terminal.

[0089] In another possible implementation, after determining that the second cursor slides out of the edge of the display screen of the first terminal, the first terminal sends first shuttling state information to the second terminal, where the first shuttling state information is used to indicate the start of shuttling.

[0090] In another possible implementation, the second terminal is further configured to send second shuttle status information to the first terminal after determining that the object is dragged out of the edge of the display screen of the second terminal, where the second shuttle status information is used to indicate the end of the shuttle.

[0091] It should be noted that the object being dragged out of the edge of the terminal (such as the second terminal) display screen described in this embodiment may mean that a partial area of ​​the object is dragged out (or overflows) the terminal display screen, or the entire area of ​​the object is dragged out (or overflows) the terminal display screen, or the cursor slides out of the edge of the terminal display screen when the object moves with the cursor on the terminal display screen. This embodiment does not make any specific restrictions here.

[0092] It can be understood that the beneficial effects that can be achieved by the cross-device object dragging device described in the third aspect and any possible implementation thereof, the cross-device object dragging device described in the fourth aspect and any possible implementation thereof, the cross-device object dragging device described in the fifth aspect, the computer-readable storage medium described in the sixth aspect, the terminal described in the seventh aspect, the computer program product described in the eighth aspect and the cross-device object dragging system described in the ninth aspect can be referred to as the beneficial effects in the first aspect or the second aspect and any possible implementation thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 A simplified schematic diagram of a system architecture provided in an embodiment of the present application;

[0094] Figure 2A A schematic diagram of the structure of a mobile phone provided in an embodiment of the present application;

[0095] Figure 2B A schematic diagram of the composition of a software architecture provided in an embodiment of the present application;

[0096] Figure 3 A schematic diagram of a flow chart of a cross-device object dragging method provided in an embodiment of the present application;

[0097] Figure 4 A schematic diagram of a coordinate system on a display screen provided in an embodiment of the present application;

[0098] Figure 5A A schematic diagram of a cross-device object dragging interface provided in an embodiment of the present application;

[0099] Figure 5B A schematic diagram of another cross-device object dragging interface provided in an embodiment of the present application;

[0100] Figure 6 A schematic diagram of another cross-device object dragging interface provided in an embodiment of the present application;

[0101] Figure 7 A schematic diagram of another cross-device object dragging interface provided in an embodiment of the present application;

[0102] Figure 8 A schematic diagram of another cross-device object dragging interface provided in an embodiment of the present application;

[0103] Fig. 9 A flowchart of another cross-device object dragging method provided in an embodiment of the present application;

[0104] Fig.10 A schematic diagram of another cross-device object dragging interface provided in an embodiment of the present application;

[0105] Fig.11 A schematic diagram of another cross-device object dragging interface provided in an embodiment of the present application;

[0106] Fig.12 A schematic diagram of another cross-device object dragging interface provided in an embodiment of the present application;

[0107] Fig.13 A schematic diagram of another cross-device object dragging interface provided in an embodiment of the present application;

[0108] Fig.14 A schematic diagram of another cross-device object dragging interface provided in an embodiment of the present application;

[0109] Fig.15 A flowchart of another cross-device object dragging method provided in an embodiment of the present application;

[0110] Fig.16 A schematic diagram of another cross-device object dragging interface provided in an embodiment of the present application;

[0111] Fig.17 A schematic diagram of the data structure of a drag event on a windows side provided in an embodiment of the present application;

[0112] Fig.18 A data structure diagram of a drag event on an Android device provided in an embodiment of the present application;

[0113] Fig.19 A schematic diagram of another cross-device object dragging interface provided in an embodiment of the present application;

[0114] Fig. 20 A flowchart of another cross-device object dragging method provided in an embodiment of the present application;

[0115] Fig.21 A schematic diagram of another cross-device object dragging interface provided in an embodiment of the present application;

[0116] Fig. 22 A schematic diagram of another cross-device object dragging interface provided in an embodiment of the present application;

[0117] Fig.23 A schematic diagram of the composition of a cross-device object dragging device provided in an embodiment of the present application;

[0118] Fig.24 A schematic diagram of another cross-device object dragging device provided in an embodiment of the present application;

[0119] Fig.25 A schematic diagram of the composition of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0120] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0121] An embodiment of the present application provides a method and device for dragging objects across devices, which can be applied to scenarios where multiple terminals are used in collaboration. In this scenario, with the help of keyboard and mouse sharing technology, the same input device (such as a mouse, touchpad) is used to control all terminals. For example, objects such as application icons or application interfaces can be dragged across devices. As another example, content (or objects) such as text or files can be transferred between multiple terminals participating in collaborative use by dragging and dropping, and users are allowed to use these terminals to process the transferred content. The method provided in this embodiment can increase the interaction between terminals when multiple terminals are used in collaboration, thereby improving the user experience.

[0122] The implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0123] Please refer to Figure 1 , is a simplified schematic diagram of a system architecture to which the above method can be applied provided by an embodiment of the present application. The system architecture can be the cross-device object dragging system in this embodiment. Figure 1 As shown, the system architecture may include at least: a first terminal 101 and a second terminal 102.

[0124] The first terminal 101 is connected to the input device 101-1 (eg Figure 1 ), or including an input device 101-1 ( Figure 1As an example, the input device 101-1 may be a mouse, a touch pad, a touch screen, etc. Figure 1 In the figure, it is taken as an example that the input device 101-1 is a mouse.

[0125] In this embodiment, the first terminal 101 and the second terminal 102 can establish a connection in a wired or wireless manner. Based on the established connection, the first terminal 101 and the second terminal 102 can be used together. In this embodiment, the wireless communication protocol used when the first terminal 101 and the second terminal 102 establish a connection in a wireless manner can be a wireless fidelity (Wi-Fi) protocol, a Bluetooth (Bluetooth) protocol, a ZigBee protocol, a Near Field Communication (NFC) protocol, etc., and can also be various cellular network protocols, which are not specifically limited here.

[0126] After the first terminal 101 is connected to the second terminal 102, using the keyboard and mouse sharing technology, the user can use a set of input devices, such as the above-mentioned input device 101-1 to control both the first terminal 101 and the second terminal 102. In other words, the user can not only use the input device 101-1 of the first terminal 101 to control the first terminal 101, but the first terminal 101 can also share its input device 101-1 with the second terminal 102, so that the user can control the second terminal 102.

[0127] For example, one or more applications are installed in the second terminal 102. The second terminal 102 can display the icon of the corresponding application on the display screen 102-1 of the second terminal 102. In this embodiment, after the first terminal 101 and the second terminal 102 are connected, the user can use the input device 101-1 to drag the icon of the application from the display screen 102-1 of the second terminal 102 to the display screen 101-2 of the first terminal 101 using the keyboard and mouse sharing technology.

[0128] For another example, the interface of the application is displayed on the display screen 102-1 of the second terminal 102. In this embodiment, after the first terminal 101 and the second terminal 102 are connected, the user can use the input device 101-1 to drag the interface of the application from the display screen 102-1 of the second terminal 102 to the display screen 101-2 of the first terminal 101 using the keyboard and mouse sharing technology.

[0129] For another example, the input device 101-1 is a mouse. After the first terminal 101 is connected to the second terminal 102, without starting screen projection, the user can use the mouse to drag the text or file content of the first terminal 101 to the second terminal 102 by using the keyboard and mouse sharing technology. The user can also use the mouse to drag the text or file content of the second terminal 102 to the first terminal 101 by dragging.

[0130] It should be noted that, under the premise of not starting screen projection, the embodiment of the present application can not only realize the dragging and dropping of content between two terminals, but also realize the dragging and dropping of content between three or more terminals. Exemplarily, when the second terminal 102 is also connected with other devices, such as the third terminal, in the embodiment of the present application, under the premise of not starting screen projection, using the keyboard and mouse sharing technology, after the user drags the text or file and other content from one terminal to another terminal, he can continue to drag the content to the third terminal by dragging. For example, continuing to take the above-mentioned input device 101-1 as a mouse as an example, under the premise of not starting screen projection, the user uses the mouse to drag the text or file and other content of the first terminal 101 to the second terminal 102 by dragging, and then can continue to drag the content to the third terminal by dragging. After the user releases the mouse, the content dragging is completed.

[0131] It should be noted that the terminals in the embodiments of the present application, such as the first terminal 101, the second terminal 102, and the third terminal, can be mobile phones, tablet computers, handheld computers, PCs, cellular phones, personal digital assistants (PDAs), wearable devices (such as smart watches), smart home devices (such as TVs), car computers, game consoles, and augmented reality (AR) and virtual reality (VR) devices, etc. This embodiment does not impose any special restrictions on the specific form of the terminal. Among them, Figure 1 In the figure, it is taken as an example that the first terminal 101 is a PC and the second terminal 102 is a mobile phone.

[0132] In this embodiment, the terminal is taken as a mobile phone. Figure 2A , is a structural diagram of a mobile phone provided in an embodiment of the present application.

[0133] like Figure 2AAs shown, the mobile phone may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0134] Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0135] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the mobile phone. In other embodiments, the mobile phone may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0136] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0137] The controller can be the nerve center and command center of the mobile phone. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0138] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0139] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0140] The charging management module 140 is used to receive charging input from the charger. While the charging management module 140 is charging the battery 142, it can also power the mobile phone through the power management module 141. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 can also receive input from the battery 142 to power the mobile phone.

[0141] The wireless communication function of the mobile phone can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0142] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the mobile phone can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0143] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on mobile phones. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0144] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0145] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), NFC, infrared technology (IR), etc., which are applied to mobile phones. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0146] In some embodiments, the antenna 1 of the mobile phone is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the mobile phone can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).

[0147] The mobile phone realizes the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0148] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the mobile phone may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0149] The mobile phone can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor, etc. In some embodiments, the mobile phone can include 1 or N cameras 193, where N is a positive integer greater than 1.

[0150] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.

[0151] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the mobile phone by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0152] The mobile phone can realize audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor.

[0153] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. When a touch operation is applied to the display screen 194, the mobile phone detects the intensity of the touch operation according to the pressure sensor 180A. The mobile phone can also calculate the touch position according to the detection signal of the pressure sensor 180A.

[0154] The gyroscope sensor 180B can be used to determine the motion posture of the mobile phone. The air pressure sensor 180C is used to measure air pressure. The magnetic sensor 180D includes a Hall sensor. The mobile phone can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. The acceleration sensor 180E can detect the magnitude of the acceleration of the mobile phone in all directions (generally three axes). The distance sensor 180F is used to measure the distance. The mobile phone can use the proximity light sensor 180G to detect that the user holds the mobile phone close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used for automatic unlocking and locking of the leather case mode and pocket mode. The ambient light sensor 180L is used to sense the brightness of the ambient light. The fingerprint sensor 180H is used to collect fingerprints. The mobile phone can use the collected fingerprint characteristics to realize fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc. The temperature sensor 180J is used to detect temperature.

[0155] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the mobile phone, which is different from the position of the display screen 194.

[0156] The bone conduction sensor 180M can obtain vibration signals. The button 190 includes a power button, a volume button, etc. The button 190 can be a mechanical button. It can also be a touch button. The motor 191 can generate a vibration prompt. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. The indicator 192 can be an indicator light, which can be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.

[0157] The SIM card interface 195 is used to connect the SIM card. The SIM card can be connected to and separated from the mobile phone by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The mobile phone can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The mobile phone interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the mobile phone uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the mobile phone and cannot be separated from the mobile phone. The methods in the following embodiments can be implemented in a mobile phone with the above-mentioned hardware structure.

[0158] Combination Figure 1 In this embodiment of the application, the software system of the first terminal 101 is a Windows system and the software system of the second terminal 102 is an Android system as an example to illustrate the software structure of the first terminal 101 and the second terminal 102. Please refer to Figure 2B , which is a schematic diagram of the composition of a software architecture provided in an embodiment of the present application.

[0159] Among them, Figure 2B As shown, the software architecture of the first terminal 101 may include: an application layer and a windows system (windows shell). In some embodiments, the application layer may include various applications installed in the first terminal 101. The applications of the application layer may directly interact with the windows system. Exemplarily, the application layer may also include a keyboard and mouse module, a drag management module, and a transmission management module. Further, the application layer may also include a window management module.

[0160] The software system of the second terminal 102 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. For example, the software system of the second terminal 102 is a layered architecture. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, Figure 2BAs shown, the second terminal 102 may include an application layer and a framework layer (framework, FWK). The application layer may include a series of application packages. For example, the application package may include settings, calculator, camera, short message, music player and other applications. The applications included in the application layer may be system applications of the second terminal 102 or third-party applications, and the embodiments of the present application do not make specific restrictions here. The application layer may also include a transmission management module and a drag management module. The application layer may also include a launcher. The application layer may also include a system interface (systemUI). The framework layer is mainly responsible for providing an application programming interface (API) and a programming framework for applications in the application layer. In some embodiments of the present invention, the framework layer may include a window manager (or window management service). Of course, the second terminal 102 may also include other layers, such as a kernel layer ( Figure 2B The kernel layer is a layer between hardware and software. The kernel layer may at least include display driver, camera driver, audio driver, sensor driver, etc.

[0161] In the scenario where multiple terminals are used in coordination, in order to improve the efficiency of use, the user can use the same input device to control these multiple terminals. Figure 1 , taking the multiple terminals including the first terminal 101 and the second terminal 102, and taking the input device 101-1 of the first terminal 101 as a mouse as an example. After the first terminal 101 and the second terminal 102 are connected, based on the above software architecture and with the help of keyboard and mouse sharing technology, the user can use the input device 101-1 of the first terminal 101 to control the first terminal 101 and the second terminal 102. For example, in some embodiments, the user can use the input device 101-1 of the first terminal 101 to drag and drop the icon of the application displayed on the second terminal 102 or the interface of the application from the second terminal 102 to the first terminal 101. For another example, in some other embodiments, without starting the screen projection, the user can use the input device 201-1 of the first terminal 201 to transfer the files or texts of the first terminal 201 to the second terminal 202 by dragging and dropping. The files or texts of the second terminal 202 can also be transferred to the first terminal 201 by dragging and dropping. That is to say, without starting screen projection, the user can use the input device 201-1 of the first terminal 201 to drag and drop the content such as files or text in the application between the first terminal 201 and the second terminal 202.

[0162] The keyboard and mouse sharing technology may refer to a technology for using an input device (such as a mouse or a touch pad) of one terminal to control other terminals.

[0163] It should be noted that, in the present embodiment, the above-mentioned drag management module may also be referred to as a drag service module. In addition, in the above-mentioned embodiment, it can be seen that both the first terminal 101 and the second terminal 102 include a transmission management module, and the communication between the first terminal 101 and the second terminal 102 can be achieved through the transmission management module. In some other embodiments, the drag management module may also have the function of communicating with other terminals, that is, the first terminal 101 and the second terminal 102 may not include a transmission management module, and the communication between them may be achieved through the drag management module, and this embodiment does not make specific restrictions here. For the convenience of description, the following embodiments are described by taking the communication between the first terminal 101 and the second terminal 102 being achieved through the transmission management module as an example.

[0164] In some embodiments, the following is combined Figure 1 and Figure 2B , taking the first terminal 101 as a PC, the second terminal 102 as a mobile phone, and the input device 101-1 as a mouse as an example, the method provided in this embodiment is described in detail in conjunction with the accompanying drawings.

[0165] Figure 3 The present invention provides a flow chart of a cross-device object dragging method. In this embodiment, the implementation process of a user using a mouse shared by a PC to drag the icon of an application displayed on a mobile phone (the icon of the application is the dragged object) from the mobile phone to the PC is introduced. Figure 3 As shown, the method may include the following S301-S313.

[0166] S301. The mobile phone establishes a connection with the PC.

[0167] In some embodiments, the mobile phone and the PC can be connected in a wired manner. For example, the mobile phone and the PC can be connected in a wired manner via a data cable.

[0168] In some other embodiments, the mobile phone and the PC can establish a connection wirelessly. There are two requirements for establishing a connection between terminals by wireless means, one is that the terminals need to know each other's connection information, and the other is that each terminal needs to have transmission capability. The connection information can be a device identifier of the terminal, such as an Internet Protocol (IP) address, a port number, or an account logged in by the terminal. The account logged in by the terminal can be an account provided by the operator to the user, such as a Huawei account. The account logged in by the terminal can also be an application account, such as a WeChat account, a Youku account, etc. The terminal has a transmission capability that can be a near-field communication capability or a long-distance communication capability. That is to say, between terminals, such as the wireless communication protocol used by the mobile phone to establish a connection with the PC can be a near-field communication protocol such as a Wi-Fi protocol, a Bluetooth protocol, or an NFC protocol, or a cellular network protocol. Take the example of establishing a connection between a mobile phone and a PC by wireless means. For example, a user can use a mobile phone to touch the NFC tag of the PC, and the mobile phone reads the connection information stored in the NFC tag, such as the connection information including the IP address of the PC. Afterwards, the mobile phone can establish a connection with the PC using the NFC protocol according to the IP address of the PC. For another example, both the mobile phone and the PC have turned on the Bluetooth function and the Wi-Fi function. The PC can broadcast Bluetooth signals to discover surrounding terminals. For example, the PC can display a list of discovered devices, which may include the identifiers of mobile phones discovered by the PC. In addition, during the device discovery process of the PC, connection information, such as IP addresses, can be exchanged with the discovered devices. Afterwards, after the PC receives the user's operation of selecting the identifier of the mobile phone in the displayed device list, the PC can use the Wi-Fi protocol to establish a connection with the mobile phone based on the IP address of the mobile phone. For another example, both the mobile phone and the PC are connected to the cellular network, and the mobile phone and the PC are logged in to the same Huawei account. The mobile phone and the PC can establish a connection based on the cellular network based on the Huawei account.

[0169] After the mobile phone and the PC are successfully connected, the two can be used together. In order to improve the efficiency of the collaborative use, the user can use a set of input devices, such as a PC mouse, to control both the PC and the mobile phone.

[0170] As an exemplary implementation, when the keyboard and mouse sharing mode of the PC is turned on, a set of input devices can be used to control both the PC and the mobile phone.

[0171] For example, in some embodiments, after the other terminal successfully establishes a connection with the PC, the PC may display a pop-up window to ask the user whether to enable the keyboard and mouse sharing mode. If the user selects to enable the keyboard and mouse sharing mode, the PC may enable the keyboard and mouse sharing mode.

[0172] After turning on the keyboard and mouse sharing mode, the PC can notify all terminals that have established a connection with itself that the keyboard and mouse sharing mode has been turned on. If the PC is connected to a mobile phone, the PC will notify the mobile phone that the keyboard and mouse sharing mode has been turned on. After receiving the notification (such as a notification message), the mobile phone can create a virtual input device, which has the same function as conventional input devices such as a mouse and a touchpad, and can be used to simulate corresponding input events on the mobile phone. For example, taking the input device as a mouse as an example, the virtual input device created by the mobile phone has the same function as a conventional mouse, which can be regarded as a mouse shared by the PC to the mobile phone, and can be used to simulate mouse events on the mobile phone side to realize the control of the mobile phone by the PC mouse. Exemplarily, taking the mobile phone operating system as an Android system as an example. The mobile phone can use the uinput capability of Linux to create a virtual input device. Among them, uinput is a kernel layer module that can simulate input devices. By writing to the / dev / uinput (or / dev / input / uinput) device, a process can create a virtual input device with specific functions. Once the virtual input device is created, it can simulate the corresponding event. Similarly, other terminals that have established a connection with the PC will also create a virtual input device based on the received notification message. It should be noted that if the operating system of the terminal that receives the notification message is an Android system, the creation of a virtual input device can be implemented using the uinput capability of Linux, or the creation of a virtual input device can be implemented using the human-computer interaction device (HID) protocol. If the operating system of the terminal that receives the notification message is an IOS system or other operating systems such as a windows system, the creation of a virtual input device can be implemented using the HID protocol. In addition, the above embodiment is described by taking the creation of a virtual input device as an example after the terminal connected to the PC receives a notification message for notifying that the keyboard and mouse sharing mode of the PC has been turned on. In some other embodiments, after the terminal connected to the PC receives the above notification message, a pop-up window can also be displayed to ask the user whether he wants to use the input device of the PC to control the device. If the user selects to use the input device of the PC to control the device, the virtual input device is created, otherwise the virtual input device is not created.

[0173] For another example, in some other embodiments, after a mobile phone is connected to a PC, the PC automatically turns on the keyboard and mouse sharing mode without the need for the user to turn it on manually. In other terminals, after the mobile phone is connected to the PC as described above, a virtual input device can also be automatically created without the need for the PC to send a notification message. Alternatively, after other terminals are connected to the PC, a pop-up window may be displayed to ask the user whether he wants to use the PC's input device to control the device. If an operation is received in which the user chooses to use the PC's input device to control the device, the virtual input device is automatically created, otherwise the virtual input device is not created.

[0174] In addition, combined Figure 1 Since the mouse is an input device of the PC, after other terminals, such as a mobile phone, are connected to the PC, generally, it is the PC that temporarily responds to the mouse operation, or the user can temporarily control the PC using the mouse. In this embodiment, after the PC turns on the keyboard and mouse sharing mode, it can also trigger other terminals that have established a connection with the PC and created a virtual input device, such as a mobile phone, to respond to the mouse operation when it is determined that the mouse shuttle condition is met. In other words, after the mouse shuttle condition is met, the user can use the mouse to control other terminals that have established a connection with the PC and created a virtual input device, such as a mobile phone.

[0175] For example, the mouse shuttle condition may be that the mouse pointer displayed on the PC screen corresponding to the mouse slides over the edge of the PC screen. That is, the user may move the mouse so that the mouse pointer displayed on the PC screen corresponding to the mouse slides over the edge of the PC screen, thereby triggering other terminals connected to the PC and creating virtual input devices to respond to the mouse operation.

[0176] As an exemplary implementation, after turning on the keyboard and mouse sharing mode, the PC can turn on input monitoring and mount a hook. Input monitoring can be used to monitor the relative displacement and coordinate position of the mouse pointer. The mounted HOOK can be used to intercept corresponding input events (or shield corresponding input events) after the mouse shuttle starts. For example, taking the input device as a mouse, the input event can be a mouse event (or shield mouse events), so that the mouse event will not be transmitted to the windows system of the PC after being received by the keyboard and mouse module of the PC. The mounted HOOK can also be used to capture intercepted input events, such as parameters in mouse events, after the mouse shuttle starts. The PC can use input monitoring to monitor the relative displacement and coordinate position of the mouse pointer, and determine whether the mouse shuttle conditions are met based on the monitored data. After determining that the mouse shuttle conditions are met, the mounted HOOK intercepts the mouse event, captures the operation parameters in the mouse event, and sends the captured operation parameters to other terminals connected to the PC that have created virtual input devices, so that the terminal uses the created virtual input device to simulate corresponding input events, such as mouse events, and then responds to them. That is, other terminals connected to the PC and having created virtual input devices respond to the mouse operation.

[0177] Of course, the interception of input events and the capture of operating parameters therein can also be achieved by other means (such as registering RAWINPUT in the PC). Alternatively, the interception of input events and the capture of operating parameters therein can also be achieved respectively by different means. For example, taking the input device as a mouse as an example, after turning on the keyboard and mouse sharing mode, the PC can mount HOOK and register RAWINPUT, wherein, after the mouse shuttle starts, the mounted HOOK can be used to intercept mouse events (or shield mouse events), and the registered RAWINPUT can be used to capture the operating parameters in the intercepted mouse events. This embodiment does not limit the specific implementation of the interception of mouse events and the capture of operating parameters therein. For the convenience of description, the following embodiments are introduced by taking the interception of input events and the capture of operating parameters therein by mounting HOOK as an example.

[0178] For example, if a user wants to drag the icon of an application displayed on a mobile phone from the mobile phone to the PC, the process of triggering the other terminal connected to the PC that has created a virtual input device, such as a mobile phone, to respond to the mouse operation may include the following S302-S304.

[0179] S302: The PC receives a mouse movement event, and displays an animation of the mouse pointer moving on a display screen of the PC according to the mouse movement event.

[0180] It should be noted that the mouse pointer described in this embodiment may also be referred to as a cursor. The cursor may be an image, which may be dynamic or static, and the style of the cursor may be different in different situations. In this embodiment, the mouse pointer is used as an example for explanation. Among them, the mouse pointer displayed on the PC display may be the second cursor in this application.

[0181] S303: The PC monitors the coordinate position of the mouse pointer on the PC display screen.

[0182] S304. The PC intercepts a mouse movement event based on the coordinate position of the mouse pointer on the PC display screen and determines that the mouse pointer slides out of the edge of the PC display screen, and sends a mouse operation parameter 1 included in the mouse movement event to the mobile phone.

[0183] In this embodiment, after the keyboard and mouse sharing mode is turned on, when the user wants to use the mouse to control other terminals connected to the PC that have created virtual input devices, such as a mobile phone, for example, when the user wants to drag the icon of an application displayed on the mobile phone from the mobile phone to the PC, the user can continue to move the mouse in the same direction to make the mouse pointer displayed on the PC screen slide over (or slide out) the edge of the PC screen, thereby triggering the mouse shuttling condition.

[0184] Exemplarily, the PC may determine the coordinate position of the mouse pointer on the PC display screen according to the initial position and relative displacement of the mouse pointer, thereby determining whether the mouse pointer slides out of the edge of the PC display screen.

[0185] The initial position of the mouse pointer may be the coordinate position of the mouse pointer on the PC display when the mouse starts to move, or the coordinate position of the mouse pointer on the PC display before the mouse starts to move. The initial position of the mouse pointer may specifically refer to the coordinate position in a coordinate system with the upper left corner of the PC display as the origin, the X axis pointing from the upper left corner to the right edge of the PC display, and the Y axis pointing from the upper left corner to the lower edge of the PC display. For example, the specific process of the PC determining whether the mouse pointer has slipped out of the edge of the PC display may be: combining Figure 4 , the PC can establish a coordinate origin with the initial coordinate position (such as Figure 4 The coordinate system is a coordinate system in which the X axis points from the coordinate origin o to the right edge of the PC display screen, and the Y axis points from the coordinate origin o to the upper edge of the PC display screen. The PC can determine the coordinate values ​​of each edge of the PC display screen in this coordinate system. The coordinate values ​​of each edge of the PC display screen in this coordinate system can be determined based on the resolution of the PC display screen and the initial position of the mouse pointer. For example, Figure 4As shown, in this coordinate system, the coordinate value of the right edge of the PC display screen on the X axis is x1, the coordinate value of the left edge on the X axis is -x2, the coordinate value of the upper edge on the Y axis is y1, and the coordinate value of the lower edge on the Y axis is -y2. After the mouse moves, the mouse will report the relative displacement of the corresponding mouse pointer to the PC. The PC can calculate the coordinate position (x, y) of the mouse pointer on the PC display screen after the mouse moves based on the relative displacement reported by the mouse. Based on the coordinate position (x, y), the PC can determine whether the mouse pointer slides out of the edge of the PC display screen. For example, if the coordinate value x of the mouse pointer on the X axis is greater than x1, it can be determined that the mouse pointer slides out of the right edge of the PC display screen. If the coordinate value x of the mouse pointer on the X axis is less than -x2, it can be determined that the mouse pointer slides out of the left edge of the PC display screen. If the coordinate value y of the mouse pointer on the Y axis is greater than y1, it can be determined that the mouse pointer slides out of the upper edge of the PC display screen. If the coordinate value y of the mouse pointer on the Y axis is less than -y2, it can be determined that the mouse pointer slides out of the lower edge of the PC display screen.

[0186] After determining that the mouse pointer slides out of the edge of the PC display screen, that is, the mouse shuttle condition is triggered, if the user continues to operate the PC's input device, such as a mouse, the PC can send the data of the operation input by the user using the PC's input device to other terminals that have created virtual input devices. For example, after the user uses the PC's input device to input an operation, the PC can intercept the corresponding input events received, such as mouse movement events, mouse press events, mouse lift events, etc., and transmit the operation parameters contained in the intercepted input events, such as mouse operation parameters, to other terminals connected to the PC that have created virtual input devices, so that the terminal can respond to the operation of the PC's input device, such as a mouse. Here, take the input event being a mouse event as an example. The operation parameters (such as mouse operation parameters) included in the input event (or mouse event) may include: mouse button flag (used to indicate whether the user has performed an operation on the mouse, such as pressing, lifting, moving or rolling the mouse wheel), coordinate information (when the user moves the mouse, used to indicate the X coordinate and Y coordinate of the mouse movement), wheel information (when the user operates the mouse wheel, used to indicate the X-axis distance and Y-axis distance of the wheel rolling), and key position information (used to indicate which of the left, middle or right buttons of the mouse the user has operated).

[0187] In some embodiments, if there is only one device connected to the PC, such as the mobile phone mentioned above, and the mobile phone has created a virtual input device, the PC can transmit the corresponding operation parameters to the mobile phone so that the mobile phone can respond to the operation of the PC's input device, such as a mouse. If there are multiple devices connected to the PC, and some or all of the multiple devices have established virtual input devices, the PC can display a list of options on the display screen of the PC when it is determined that the mouse shuttle condition is triggered. The list of options includes the identifier of the device that has created the virtual input device among the devices connected to the PC (such as the identifier of the mobile phone mentioned above). The PC can determine the device that responds to the operation of the PC's input device according to the user's selection. If the user selects the identifier of the mobile phone mentioned above, the PC can send the corresponding operation parameters to the mobile phone so that the mobile phone can respond to the operation of the PC's input device. After the mobile phone receives the corresponding operation parameters, it can simulate the corresponding input device, such as a mouse event, and make a corresponding response, that is, the mobile phone can respond to the operation of the PC's input device. In this embodiment, after completing the creation of the virtual input device, the device connected to the PC can send an indication message to the PC that the virtual input device has been successfully created. According to the received indication message, the PC can obtain which devices among the devices connected to the PC have successfully created virtual input devices, and based on this, display the above list options.

[0188] In some other embodiments, the shuttle relationship can be preconfigured. If there are multiple devices connected to the PC, and some or all of the multiple devices have established virtual input devices, it can be determined according to the preconfigured shuttle relationship to which device the mouse shuttles, that is, it can be determined which device responds to the operation of the PC input device. For example, the multiple devices connected to the PC include the above-mentioned mobile phone, and the mobile phone creates a virtual input device. The preconfigured shuttle relationship is that the mouse pointer slides out from the left side (or the left edge) of the PC display screen, and the mouse shuttles to the mobile phone. Then, when the user moves the mouse so that the mouse pointer slides over the left edge of the PC display screen, the PC can not only determine that the mouse shuttle starts, but also determine that the mouse shuttles to the mobile phone, that is, the PC can send the corresponding operation parameters to the mobile phone, so that the mobile phone responds to the operation of the PC input device. Of course, if there is only one device connected to the PC, and the device creates a virtual input device, it can also be determined according to the preconfigured shuttle relationship whether the mouse shuttles to the device. For example, the preconfigured shuttle relationship is that the mouse pointer slides out from the left edge of the PC display screen, and the mouse shuttles to the mobile phone. However, after the user moves the mouse, the mouse pointer slides over the right edge of the PC display screen, then it can be determined that the mouse does not shuttle to the mobile phone. In some other embodiments, the device to which the mouse shuttles can be determined by identifying the device position. For example, taking the input device as a mouse as an example, the user presses and moves the mouse so that the mouse pointer slides over the left edge of the PC display screen, then the location of the device located around the PC can be identified using positioning technologies such as Bluetooth, ultra-wideband (UWB), and ultrasound. If the PC recognizes that the left side of the PC is a mobile phone, it can be determined that the mouse shuttles to the mobile phone.

[0189] Among them, the shuttle relationship can be configured in advance by the user through a configuration file, or a configuration interface for configuring the shuttle relationship can be provided to the user, and the user can configure the shuttle relationship in advance through the configuration interface. For example, take the example of a user configuring the shuttle relationship of a mobile phone through an interface. The PC receives the user's operation of opening the configuration interface and displays the configuration interface, which includes the PC's logo (such as the PC's icon) and the mobile phone's logo (such as the mobile phone's icon). The user can configure the shuttle relationship by dragging these two logos. As an example, if the user places the mobile phone's logo to the left of the PC's logo, the PC can determine that when the mouse pointer slides over the left edge of the PC display screen, the mouse shuttles to the mobile phone. If the user places the mobile phone's logo to the right of the PC's logo, the PC can determine that when the mouse pointer slides over the right edge of the PC display screen, the mouse shuttles to the mobile phone. When there are multiple devices, the shuttle relationship of each device can be configured in a pre-configured manner. The following embodiments are all described by taking the determined mouse shuttle to the mobile phone as an example. It should be noted that for the two implementations of determining which device the mouse shuttles to based on the pre-configured shuttle relationship and identifying the device position, the above S301 can be executed before the mouse shuttle is triggered or after the mouse shuttle is triggered, and this embodiment does not make any specific restrictions here.

[0190] For example, when a user uses the input device of a PC to input a movement operation, such as when the user moves the mouse, the keyboard and mouse module of the PC may receive a corresponding input event, such as a movement event, which may be referred to as a mouse movement event. Since the mouse shuttle condition has not yet been triggered, the HOOK will not intercept the input event, that is, it will not intercept the mouse movement event, and the mouse movement event will be transmitted to the Windows system of the PC. Based on the received mouse movement event, the Windows system of the PC may draw an animation of the movement of the mouse pointer and display it on the display screen of the PC. For example, Figure 5A As shown, as the mouse 401 moves, the PC displays an animation of the movement of the mouse pointer 403 on the display screen 402 of the PC, as shown in FIG. Figure 5A The movement track of the mouse pointer 403 shown in FIG. 4 is shown as track 404 .

[0191] As described in S301, after the keyboard and mouse sharing mode is turned on, the PC will turn on input monitoring and mount HOOK. In the process of the mouse pointer moving on the PC display screen, the keyboard and mouse module of the PC can use input monitoring to monitor the real-time coordinate position of the mouse pointer on the PC display screen. When the keyboard and mouse module of the PC determines that the mouse pointer slides over the edge of the PC display screen based on the real-time coordinate position of the mouse pointer on the PC display screen, it can be determined that the above-mentioned mouse shuttle condition is met, indicating that the user wants to use the mouse to control other terminals. The keyboard and mouse module of the PC can determine that the mouse shuttle starts.

[0192] After the keyboard and mouse module of the PC determines that the mouse shuttle has started, if the user operates the input device of the PC, such as a mouse, the keyboard and mouse module of the PC will use HOOK to intercept the input device, such as a mouse event, and use HOOK to capture the operation parameters in the intercepted input event, such as the mouse operation parameters, and then transmit the operation parameters to the mobile phone through the established connection through the transmission management module of the PC, so that the mobile phone can make a corresponding response. For example, in combination with Figure 5A In the example shown, taking the input device as a mouse, after the mouse pointer slides over the edge of the PC display screen, the user continues to move the mouse in the same direction. The keyboard and mouse module of the PC can receive movement events, such as mouse movement events. Since the mouse shuttle has started, the keyboard and mouse module of the PC can use HOOK to intercept (or shield) it so that the mouse movement event will not be sent to the windows system of the PC, so that the PC will not respond to the received mouse movement event. The keyboard and mouse module of the PC can also use HOOK to capture the operation parameters in the mouse movement event, such as the mouse operation parameter (such as called mouse operation parameter 1, which can be the third operation parameter in this application), and send the mouse operation parameter 1 to the mobile phone through the established connection through the transmission management module of the PC. For example, as an example, the mouse operation parameter 1 corresponding to the mouse movement event can be: a mouse button flag for indicating that the user has moved the mouse, coordinate information for indicating the X coordinate and Y coordinate of the mouse movement, scroll wheel information (value is empty) and key position information (value is empty).

[0193] In addition, after the keyboard and mouse module of the PC determines that the mouse shuttle has started, the keyboard and mouse module of the PC can send shuttle status information indicating that the mouse has started to shuttle to the mobile phone through the transmission management module of the PC through the established connection (the shuttle status information can be the first shuttle status information in this application). After the mobile phone receives the information, it can simulate a mouse pointer and display the mouse pointer on the display screen of the mobile phone (wherein the mouse pointer displayed on the display screen of the mobile phone can be the first cursor in this application). For example, after the transmission management module of the mobile phone receives the information, it transmits the information to the drag management module and launcher of the mobile phone, so that the drag management module of the mobile phone can create a mouse pointer and hand it over to the launcher of the mobile phone to display the mouse pointer. The keyboard and mouse module of the PC can also hide the mouse pointer displayed on the PC display. Give the user the visual effect of the mouse pointer shuttling from the PC to the mobile phone.

[0194] S305: The mobile phone receives the mouse operation parameter 1, simulates a mouse movement event according to the mouse operation parameter 1, and displays an animation of the movement of the mouse pointer on the display screen of the mobile phone according to the mouse movement event.

[0195] After the mobile phone receives the operation parameter, such as the mouse operation parameter 1, a movement event, such as a mouse movement event, can be simulated according to the mouse operation parameter 1. In response to the mouse movement event, the mobile phone can draw an animation of the mouse pointer moving and display it on the display screen of the mobile phone.

[0196] Generally, for applications installed in a mobile phone, the mobile phone can display icons of these applications on the desktop (or home screen) of the mobile phone. In combination with the description in S304, the purpose of the user triggering the mouse pointer to shuttle from the display screen of the PC to the display screen of the mobile phone is to drag the icon of the application displayed on the display screen of the mobile phone from the mobile phone to the PC. Therefore, after the user shuttles the mouse pointer from the display screen of the PC to the display screen of the mobile phone by moving the mouse, the user will continue to move the mouse, that is, the user will use the mouse of the PC to input the operation of moving the mouse pointer on the mobile phone, until the mouse pointer moves to the position of the icon of the application that the user wants to drag displayed on the display screen of the mobile phone.

[0197] It should be noted that the application described in this embodiment can be an embedded application (i.e., a system application of the mobile phone, such as a calculator, camera, settings, gallery, etc.), or a downloadable application (such as a browser, weather, email, etc.). An embedded application is an application provided as part of a mobile phone implementation. A downloadable application is an application that can provide its own Internet protocol multimedia subsystem (IMS) connection. The downloadable application can be an application pre-installed in the mobile phone or can be a third-party application downloaded and installed in the mobile phone by the user.

[0198] Exemplarily, after the transmission management module of the mobile phone receives an operation parameter, such as mouse operation parameter 1, the transmission management module of the mobile phone may transmit the mouse operation parameter 1 to the drag management module of the mobile phone. Since the operating systems of the PC and the mobile phone are different, there are differences in the key values ​​of the mouse operation parameters in their input events, such as mouse events. Therefore, the drag management module of the mobile phone can convert the received key code of the mouse operation parameter 1 into a key code that can be recognized by the mobile phone according to a preset mapping relationship. Afterwards, the mobile phone can use the created virtual input device to simulate an input device that can be recognized by the mobile phone according to the mouse operation parameter 1 after the key code is converted, such as the corresponding mouse event, that is, it can simulate a movement event, such as a mouse movement event. The drag management module of the mobile phone can draw an animation of the movement of the mouse pointer based on the simulated mouse movement event, and let the mobile phone's launcher display the animation of the movement of the mouse pointer on the mobile phone's display screen. For example, if Figure 5BAs shown, the icon of the application that the user wants to drag is a calculator icon 504 displayed on the mobile phone display screen. As the mouse 401 moves, the mobile phone displays an animation of the movement of the mouse pointer 502 on the mobile phone display screen 501, such as the movement track of the mouse pointer 502 is shown as track 503. That is, as the mouse 401 moves, the mouse pointer 502 can move along the track 503 to the position of the calculator icon 504.

[0199] After the mouse pointer moves to the position of the object to be dragged, such as the icon of an application, the user can input a drag operation through the mouse of the PC so that the mobile phone can drag the icon of the application according to the drag operation. The drag operation can be an operation for indicating the initiation of dragging the object, such as the icon of the application. The drag operation may include one operation or multiple operations. For example, the drag operation may include two operations, namely a press operation and a move operation. Taking the input device as a mouse as an example, the press operation may be a mouse press operation, and the move operation may be a mouse move operation. The process of the user using the mouse of the PC to drag the icon of the application on the mobile phone may include the following S306-S309.

[0200] S306: The PC receives a mouse press event, intercepts the mouse press event, and sends the mouse operation parameter 2 included in the mouse press event to the mobile phone.

[0201] Among them, the mouse operation parameter 2 may be the first operation parameter in this application.

[0202] S307: The mobile phone receives the mouse operation parameter 2, simulates a mouse press event according to the mouse operation parameter 2, and obtains application information according to the mouse press event.

[0203] Among them, a press event, such as a mouse press event, is an event received by the PC after the user presses the input device of the PC, such as a mouse, when the mouse pointer moves to the display position of the application icon. It should be noted that, here, the input device is a mouse as an example, and the "press operation on the mouse, or the mouse press operation" can be the press operation of the left mouse button or the press operation of the right mouse button, and this embodiment does not make specific restrictions here.

[0204] Combination Figure 5B As described in S305, the user wants to change the icon of the application displayed on the mobile phone display (eg, Figure 5BThe calculator icon 504 in the example shown is dragged from the mobile phone to the PC. Taking the input device as a mouse as an example, when the user moves the mouse pointer to the position of the calculator icon 504 by moving the mouse, the user can press the mouse (such as the left mouse button). The keyboard and mouse module of the PC can receive a press event, such as a mouse press event. Since the mouse press event is received by the PC after the mouse shuttle starts, the keyboard and mouse module of the PC will use HOOK to intercept (or shield) it so that the mouse press event will not be sent to the windows system of the PC, so that the PC will not respond to the received mouse movement event. The keyboard and mouse module of the PC can also use HOOK to capture the operation parameters of the mouse press event, such as the mouse operation parameter, such as the mouse operation parameter 2, and send the captured mouse operation parameter 2 to the mobile phone through the established connection through the transmission management module of the PC. Among them, the mouse operation parameter 2 may include: a mouse button flag for indicating that the user has pressed the mouse, coordinate information (the value is empty), wheel information (the value is empty) and key position information for indicating that the user has operated the left button of the mouse.

[0205] The transmission management module of the mobile phone can receive the corresponding mouse operation parameter 2 and send the mouse operation parameter 2 to the drag management module of the mobile phone. After receiving the mouse operation parameter 2, the drag management module of the mobile phone converts the key code of the received mouse operation parameter 2 into a key code that can be recognized by the mobile phone according to the preset mapping relationship, and can use the created virtual input device to simulate a press event that can be recognized by the mobile phone according to the mouse operation parameter 2 after the key code is converted, such as a mouse press event. Afterwards, the drag management module of the mobile phone can determine that the user wants to drag the icon of the application (such as the calculator icon) at the current position of the mouse pointer according to the current displayed position of the mouse pointer. For example, after the keyboard and mouse shuttle starts (such as the mobile phone receives the shuttle status information from the PC for indicating that the mouse starts to shuttle), the mobile phone can register a listener for the coordinate position of the mouse pointer. Through the listener, the mobile phone can monitor the coordinate position of the mouse pointer on the mobile phone display in real time. That is, the drag management module of the mobile phone can use the listener to determine the current coordinate position of the mouse pointer. Afterwards, the mobile phone can obtain information about the application. For example, the launcher of the mobile phone can obtain the application information and send the obtained application information to the drag management module of the mobile phone. The drag management module of the mobile phone saves the received application information. The application information may include one or more of the following information: the name of the application, the icon of the application, the package name of the application, the class name of the application, the size of the application icon, and the display position of the application icon on the mobile phone display screen.

[0206] In addition, in this embodiment, when the mobile phone simulates a press event, such as a mouse press event, that is, determines that the user performs a press operation on the application icon, such as a mouse press operation, the drag management module of the mobile phone can also create an icon identical to the icon of the application (such as a drag icon, which can be a drag object in this application) for subsequent dragging. Among them, the drag management module of the mobile phone can make appropriate adjustments to the transparency of the drag icon, or it can not make adjustments. Optionally, for the icon of the application displayed on the mobile phone display (such as the original icon), the drag management module of the mobile phone can perform a hidden process, or it can not perform a hidden process, and the mobile phone launcher continues to display it on the mobile phone display.

[0207] S308. The PC receives a mouse movement event, intercepts the mouse movement event, and sends the mouse operation parameter 3 included in the mouse movement event to the mobile phone.

[0208] Among them, the mouse operation parameter 3 may be the second operation parameter in this application.

[0209] S309: The mobile phone receives the mouse operation parameter 3, simulates a mouse movement event according to the mouse operation parameter 3, and displays an animation of the application icon following the movement of the mouse pointer on the display screen of the mobile phone as a response to the mouse press event and the mouse movement event.

[0210] When the user wants to drag the icon of the application, in the description of S307, after the user presses the mouse, he can keep his finger and move the mouse while continuing to press the mouse. At this time, the keyboard and mouse module of the PC can receive movement events, such as mouse movement events. Similarly, since the mouse movement event is received by the PC after the mouse shuttle starts, the keyboard and mouse module of the PC will use HOOK to intercept (or block) it so that the mouse movement event will not be sent to the windows system of the PC, so that the PC will not respond to the received mouse movement event. The keyboard and mouse module of the PC can also use HOOK to capture the operation parameters contained in the mouse movement event, such as mouse operation parameters, such as mouse operation parameters 3, and send it to the mobile phone through the established connection through the transmission management module of the PC.

[0211] The transmission management module of the mobile phone can receive the mouse operation parameter 3 and send the mouse operation parameter 3 to the drag management module of the mobile phone. After receiving the mouse operation parameter 3, the drag management module of the mobile phone converts the mouse operation parameter 3 into a key code, and then uses the created virtual input device to simulate the corresponding movement event, such as a mouse movement event. Afterwards, in combination with the simulated press event and movement event (such as a mouse press event and a mouse movement event), the drag management module of the mobile phone can initiate the dragging of the application icon, and can also draw an animation of the application icon following the movement of the mouse pointer, and hand it over to the mobile phone's launcher for display on the mobile phone's display screen.

[0212] For example, combined with Figure 5B In the example of FIG. 1 , after the user presses the mouse pointer at the position of the calculator icon, the user does not lift his finger and continues to move the mouse while pressing the mouse. Then, as Figure 6 As shown, as the mouse 401 moves, the mobile phone displays an animation of a calculator icon 602 following the movement of the mouse pointer 603 on the display screen 601 of the mobile phone, such as the trajectory of the calculator icon 602 following the movement of the mouse pointer 603 as shown in trajectory 604. In some embodiments, the calculator icon 602 that follows the movement of the mouse pointer 603 can be the drag icon described in the description of S307, that is, an icon created (or copied) by the mobile phone that is the same as the calculator icon displayed on the desktop. It is also understandable that when the mobile phone does not hide the original calculator icon, after the drag icon is dragged, the display screen of the mobile phone displays not only the original calculator icon, but also the drag icon that moves with the mouse pointer 603. It should be noted that in some embodiments, as the mouse moves, only the application icon can move with it, that is, only the application icon is dragged. In some other embodiments, as the mouse moves, the application icon and the application name are dragged together, that is, the application icon and the application name are dragged together (such as Figure 6 In the embodiment where the application icon and the application name are dragged together, when the mobile phone creates the drag icon, it also needs to create a name that is the same as the name of the dragged application for dragging.

[0213] In this embodiment, while the drag icon is being dragged, the mobile phone may not search for a new location for the icon or turn pages of the desktop, but instead executes the following S310-S311 so that the drag icon can be dragged onto the display screen of the PC.

[0214] S310. The mobile phone monitors the coordinate position of the mouse pointer on the mobile phone display screen.

[0215] S311. The mobile phone sends shuttle status information indicating the end of mouse shuttle to the PC when determining that the mouse pointer slides out of the edge of the mobile phone display screen according to the coordinate position of the mouse pointer on the mobile phone display screen, and sends an application icon to the PC.

[0216] In the process of the application icon moving with the mouse pointer on the display screen of the mobile phone, the mobile phone can determine whether the icon of the dragged application is dragged out of the edge of the mobile phone display screen. In this embodiment, the application icon being dragged out of the edge of the mobile phone display screen can be that a part of the application icon is dragged out (or overflows) the mobile phone display screen (that is, the proportion of the application icon overflowing the display screen area is greater than a preset threshold), or it can be that the entire area of ​​the application icon is dragged out (or overflows) the mobile phone display screen, or it can be that when the application icon moves with the mouse pointer on the mobile phone display screen, the mouse pointer slides out of the edge of the mobile phone display screen. This embodiment does not make specific restrictions here. The following is an example of determining whether the dragged application icon is dragged out of the edge of the mobile phone display screen, specifically determining whether the mouse pointer slides out of the edge of the mobile phone display screen.

[0217] In the process of the application icon moving with the mouse pointer on the display screen of the mobile phone, the mobile phone (such as the transmission management module of the mobile phone) can monitor the real-time coordinate position of the mouse pointer on the display screen of the mobile phone (as described in S307, the real-time coordinate position of the mouse pointer can be obtained using a registered listener). Exemplarily, the mobile phone can determine the coordinate position of the mouse pointer on the display screen of the mobile phone according to the initial position and relative displacement of the mouse pointer, thereby determining whether the mouse pointer slides out of the edge of the display screen of the mobile phone. Among them, the initial position of the mouse pointer can be the coordinate position of the mouse pointer on the display screen of the mobile phone when the mouse starts to move, or the coordinate position of the mouse pointer on the display screen of the mobile phone before the mouse starts to move. The initial position of the mouse pointer can specifically refer to the coordinate position in the coordinate system with the upper left corner of the display screen of the mobile phone as the coordinate origin, the X axis points from the upper left corner to the right edge of the display screen of the mobile phone, and the Y axis points from the upper left corner to the lower edge of the display screen of the mobile phone. The specific implementation of the mobile phone determining that the mouse pointer slides out of the edge of the display screen of the mobile phone is similar to the specific implementation of the PC determining that the mouse pointer slides out of the edge of the display screen of the PC, and will not be described in detail here.

[0218] When the transmission management module of the mobile phone determines that the mouse pointer slides over the edge of the mobile phone display screen based on the real-time coordinate position of the mouse pointer on the mobile phone display screen, it indicates that the user wants to use the mouse to control other devices. As described in S304, if the mobile phone has established a connection with only one device, the PC, it indicates that the user wants to use the mouse to control the PC. If the mobile phone has established connections with multiple devices, the mobile phone can display a list of options, which includes the identifiers of all devices connected to the mobile phone for the user to select the device they want to control with the mouse. If the user selects the identifier of the PC, it indicates that the user wants to use the mouse to control the PC. Alternatively, the shuttle relationship can be pre-configured in the mobile phone to determine which device the mouse shuttles to, that is, to determine which device responds to the mouse operation. The specific description of the configuration and application of the shuttle relationship is similar to the description of the corresponding content in the above embodiment, and will not be repeated here.

[0219] When it is determined that the user wants to use the mouse to control the PC, the transmission management module of the mobile phone can determine that the mouse shuttle is finished. At this time, the transmission management module of the mobile phone can send shuttle status information indicating the end of the mouse shuttle to the PC through the established connection (the shuttle status information can be the second shuttle status information in the present application). The transmission management module of the mobile phone can also indicate the end of the mouse shuttle to the drag management module of the mobile phone, so that the drag management module of the mobile phone can send drag data to the PC through the established connection through the transmission management module of the mobile phone, such as the drag data can include the icon of the application. Among them, before sending the icon of the application to the PC, the mobile phone can also serialize the icon of the application, that is, the icon of the application sent by the mobile phone to the PC can be the data obtained after serialization. If the name of the application and the icon of the application are dragged together, the mobile phone also needs to send the name of the application to the PC.

[0220] In addition, when the mouse pointer slides over the edge of the mobile phone display screen, that is, when it is determined that the mouse shuttle is completed, the dragging of the application icon on the mobile phone display screen is completed, and the drag management module of the mobile phone can also indicate the end of the mouse shuttle to the launcher of the mobile phone, so that the launcher of the mobile phone can display the drag end animation on the mobile phone display screen, such as showing that the drag icon moves from the edge of the mobile phone display screen along a trajectory (such as Figure 6 The direction is opposite to the direction of the track 604) and the animation of returning to the original position. The mobile phone does not re-layout the icons of the application displayed on the mobile phone display.

[0221] It should be noted that the above embodiments are described by taking the example that after determining that the mouse shuttle has ended, the mobile phone sends the drag data, such as the icon of the above-mentioned application, to the PC. In some other embodiments, the mobile phone may also temporarily cache the drag data, such as the information of the object (such as the icon of the application) when the drag starts. After determining that the mouse shuttle has ended, the mobile phone sends the shuttle status information indicating the end of the mouse shuttle to the PC. After receiving the shuttle status information, the PC may send a message to the mobile phone requesting the drag data, and after receiving the message, the mobile phone sends the drag data to the PC.

[0222] S312: The PC receives a mouse movement event, and displays an animation of the application icon following the movement of the mouse pointer on the display screen of the PC according to the mouse movement event and the received application icon.

[0223] After the transmission management module of the PC receives the shuttle status information from the mobile phone indicating the end of the mouse shuttle, it can indicate the end of the mouse shuttle to the drag management module of the PC and the keyboard and mouse module of the PC. The PC can determine that the user needs to switch back to the PC to respond to the mouse operation. After that, the PC can uninstall HOOK, that is, cancel the interception of input events, such as mouse events. Thereafter, when the user operates the input device of the PC, the keyboard and mouse module of the PC will not intercept the received input events, but will send the received input events to the windows system of the PC so that the windows system of the PC can respond to the input events. The keyboard and mouse module of the PC can also re-display the mouse pointer on the PC display.

[0224] Exemplary, take the input device as a mouse. The user continues to press the mouse and move the mouse at the same time. The PC can receive movement events, such as mouse movement events. Since the HOOK has been uninstalled at this time, the keyboard and mouse module of the PC will send the received mouse movement event to the windows system of the PC so that it can respond accordingly. For example, the PC can draw the application icon based on the mouse movement event and the received application icon, and display the application icon in the form of a window. The PC will also draw an animation of the application icon moving with the mouse pointer, and display the animation of the application icon following the movement of the mouse pointer on the display screen of the PC. If the application name and the application icon are dragged together, the PC must also draw the application name and display it in the form of a window, which will also move with the movement of the mouse pointer. For example, combined with Figure 6 ,like Figure 7 As shown, as the mouse 401 moves, the PC displays an animation of the calculator icon 702 following the movement of the mouse pointer 703 on the PC display screen 701, such as the trajectory of the calculator icon 702 following the movement of the mouse pointer 703 as shown in trajectory 704. That is, in combination with Figure 6 and Figure 7As shown, by long pressing the mouse and moving the mouse, the user can visually drag the icon of the application (such as the calculator icon) displayed on the mobile phone display from the mobile phone display to the PC display. Of course, if the mobile phone also performs serialization processing on the application icon before sending it to the PC, the PC can obtain the application icon by deserializing the data after receiving it.

[0225] In addition, in S311, the mobile phone may also send the size of the application icon obtained in S307 and the display position of the application icon on the mobile phone display screen to the PC. Based on these data, the PC may determine the size and initial position of the window used to display the application icon using preset rules. As an example, the preset rule may be that the size of the window is the same as the size of the application icon obtained in S307, and the initial position of the window is the same as the display position of the application icon obtained in S307 on the mobile phone display screen (e.g., the display position of the application icon obtained in S307 on the mobile phone display screen is the lower right corner of the mobile phone display screen, then the initial position of the window is the lower right corner of the PC display screen). Alternatively, the size of the window is twice the size of the application icon obtained in S307, and the initial position of the window is the same as the display position of the application icon obtained in S307 on the mobile phone display screen, etc. The present application embodiment does not specifically limit this rule.

[0226] Among them, S312 is an optional step.

[0227] S313: The PC receives a mouse up event, and displays an application interface on the PC display screen according to the mouse up event.

[0228] After the user drags the icon of the application that he wants to drag (such as the icon of a calculator) to the display screen of the PC, the user can input a drag-and-release operation to end dragging the icon of the application. For example, the user can lift his finger (that is, the drag-and-release operation can be an operation of lifting a finger, or stopping the long press operation of the mouse). The user can also stop moving the mouse. The keyboard and mouse module of the PC can receive the lift event, such as a mouse lift event. Similarly, since the HOOK has been uninstalled at this time, the keyboard and mouse module of the PC will send the received lift event, such as a mouse lift event, to the windows system of the PC for response. For example, in response to the mouse lift event, the PC can display the interface of the dragged application at the position where the mouse pointer stops moving on the display screen of the PC. For example, in combination with Figure 7 For example, Figure 8As shown in FIG. 8 , after the user drags the calculator icon to the display screen of the PC, the user lifts his finger and stops moving the mouse. At this time, in response, the PC may display the calculator interface 802 on the PC display screen 801. In addition, for the mobile phone, after the user lifts his finger and stops moving the mouse, the interface displayed on the mobile phone may not change, such as Figure 8 As shown in (a) in FIG. 8 , the desktop 803 is still displayed, and the mobile phone can also display the interface of the calculator, such as Figure 8 As shown in (b) , the mobile phone displays a calculator interface 804 .

[0229] As an example, the specific implementation of the PC displaying the interface of the application on the display screen of the PC according to the mouse lift event can be: after the keyboard and mouse module of the PC receives the lift event, such as the mouse lift event, the keyboard and mouse module of the PC can send a message to the mobile phone through the transmission management module of the PC and the connection established with the mobile phone. The message can be a request message for requesting the interface of the application, or the message can be an indication message for indicating the end of dragging the icon of the application. After the mobile phone receives the message, it can open the application according to the package name and class name of the application in the information of the application obtained in S307, and send the opened interface of the application to the PC through the connection established with the PC, so that the PC can display the interface of the application on the display screen of the PC. For example, the PC and the mobile phone can use wireless projection (Miracast) to realize the display of the interface of the application on the display screen of the PC. That is, after the mobile phone receives the above message, all layers of the interface of the obtained application can be integrated into a video stream (or called screen recording data) and encoded and sent to the PC through the real-time streaming protocol (RTSP) protocol. After receiving the video stream, the PC can decode and play it to realize the display of the interface of the application on the display screen of the PC. For another example, the mobile phone can use the distributed multi-media protocol (DMP) technology to realize the display of the application interface on the PC display screen. For example, after receiving the above message, the mobile phone can create a virtual display (VirtualDisplay). After that, the mobile phone can move the obtained drawing of the application interface to the VirtualDisplay. The mobile phone can perform screen recording processing on the application interface drawn on the VirtualDisplay. In this way, the mobile phone can obtain the screen recording data of the application interface. The mobile phone can encode the obtained screen recording data and send it to the PC. The PC can receive the corresponding data and obtain the screen recording data after decoding the data. The PC can realize the display of the application interface on the PC display screen based on the screen recording data obtained after decoding.

[0230] The method provided in this embodiment allows, in a scenario where multiple terminals are used in collaboration, when the same input device, such as a mouse, is used to control all terminals. The user can drag the icon of an application displayed on a terminal from the terminal to other terminals by moving the mouse, thereby increasing the interaction between the terminals when multiple terminals are used in collaboration and improving the user experience.

[0231] Fig. 9 This is a flowchart of another cross-device object dragging method provided by an embodiment of the present application. In this embodiment, the implementation process of a user using a mouse shared by a PC to drag the interface of an application displayed on a mobile phone (the interface of the application is the dragged object) from the mobile phone to the PC is introduced. Fig. 9 As shown, the method may include the following S901-S913.

[0232] S901. Establish a connection between the mobile phone and the PC.

[0233] S902: The PC receives a mouse movement event, and displays an animation of the mouse pointer moving on a display screen of the PC according to the mouse movement event.

[0234] S903: The PC monitors the coordinate position of the mouse pointer on the PC display screen.

[0235] S904. The PC intercepts a mouse movement event based on the coordinate position of the mouse pointer on the PC display screen and determines that the mouse pointer slides out of the edge of the PC display screen, and sends a mouse operation parameter 1 included in the mouse movement event to the mobile phone.

[0236] It should be noted that for the description of other related contents in S901-S904, please refer to Figure 3 The description of the corresponding contents in S301-S304 in the illustrated embodiment is not repeated here.

[0237] S905. The mobile phone receives the mouse operation parameter 1, simulates a mouse movement event according to the mouse operation parameter 1, and displays an animation of the movement of the mouse pointer on the display screen of the mobile phone according to the mouse movement event.

[0238] Among them, mouse operation parameter 1 is the third operation parameter in this application.

[0239] After the mobile phone receives the mouse operation parameter 1, it can simulate a movement event, such as a mouse movement event, according to the mouse operation parameter 1. In response to the mouse movement event, the mobile phone can draw an animation of the mouse pointer moving and display it on the display screen of the mobile phone. In other words, the user can trigger the mouse pointer to shuttle from the display screen of the PC to the display screen of the mobile phone by moving the mouse.

[0240] For example, the purpose of a user triggering the mouse pointer to shuttle from the PC display screen to the mobile phone display screen is to drag the interface of the application currently displayed on the mobile phone display screen from the mobile phone to the PC. In this embodiment, the operation of triggering the windowing of the application interface can be a predefined operation, such as the operation can be a long press operation on the status bar. Therefore, after the user shuttles the mouse pointer from the PC display screen to the mobile phone display screen by moving the mouse, the user will continue to move the mouse until the mouse pointer moves to the position of the status bar displayed on the mobile phone display screen.

[0241] For example, a user wants to drag the calculator interface displayed on a mobile phone to the display screen of a PC. Fig.10 As shown, the display screen of the mobile phone currently displays a calculator interface 1001. After the user moves the mouse 1002 to trigger the mouse pointer to shuttle from the display screen of the PC to the display screen of the mobile phone, the user will continue to move the mouse 1002. As the mouse 1002 moves, the mobile phone displays an animation of the movement of the mouse pointer 1003 on the display screen of the mobile phone, such as the movement track of the mouse pointer 1003 is shown as track 1004. That is, as the mouse 1002 moves, the mouse pointer 1003 can move along the track 1004 to the position of the status bar 1005 displayed on the mobile phone.

[0242] S906. The PC receives a mouse press event and a mouse lift event, intercepts the mouse press event and the mouse lift event, and sends a mouse operation parameter 2 of the mouse press event and a mouse operation parameter 3 of the mouse lift event to the mobile phone.

[0243] Among them, the mouse operation parameter 2 is the fourth operation parameter in this application, and the mouse operation parameter 3 is the fifth operation parameter in this application.

[0244] S907. The mobile phone receives mouse operation parameter 2 and mouse operation parameter 3, simulates a mouse press event according to mouse operation parameter 2, simulates a mouse lift event according to mouse operation parameter 3, displays a window on the mobile phone display screen according to the mouse press event and the mouse lift event, the window includes a screenshot of the application interface, and obtains window information.

[0245] Combination Fig.10 As described in S905, the user wants to change the interface of the application displayed on the mobile phone display (eg, Fig.10The calculator interface 1001 in the example shown is dragged from the mobile phone to the PC, and the operation that triggers the windowing of the application interface is a long press operation on the status bar as an example. When the user moves the mouse pointer to the position of the status bar by moving the mouse, the user can press the mouse (such as the left mouse button) and lift the finger after a certain period of time. The keyboard and mouse module of the PC can receive press events (such as mouse press events) and lift events (such as mouse lift events) and intercept them. The keyboard and mouse module of the PC can also capture the operation parameters of the intercepted mouse press events and mouse lift events (such as mouse operation parameters 2 and mouse operation parameters 3) through the transmission management module of the PC and send them to the mobile phone through the established connection.

[0246] The transmission management module of the mobile phone can receive the mouse operation parameter 2 and the mouse operation parameter 3, and send them to the drag management module of the mobile phone. After the drag management module of the mobile phone receives the mouse operation parameter 2 and the mouse operation parameter 3, according to the preset mapping relationship, the key code of the received mouse operation parameter 2 is converted into a key code that the mobile phone can recognize, and after the key code of the received mouse operation parameter 3 is converted into a key code that the mobile phone can recognize, the created virtual input device can be used to simulate a press event, such as a mouse press event, according to the mouse operation parameter 2 after the key code is converted, and the created virtual input device can be used to simulate a lift event, such as a mouse lift event, according to the mouse operation parameter 3 after the key code is converted. Afterwards, the drag management module of the mobile phone can determine whether the user wants to drag the interface of the application currently displayed on the mobile phone (such as the interface of the calculator) according to the current displayed position of the mouse pointer. In response to the mouse press event and the mouse lift event, the systemUI of the mobile phone sends a status bar event to the launcher of the mobile phone. After receiving the status bar event, the launcher of the mobile phone can window the interface of the currently displayed application, that is, display a window on the display screen of the mobile phone, and the window includes the interface of the application, such as a screenshot of the application interface (such as a screenshot of the calculator interface). The size of the window can be a preset value. In addition, the mobile phone (such as the launcher of the mobile phone) can also perform background blur processing. In this embodiment, the background can be the interface of the application or the desktop of the mobile phone, and this embodiment is not limited here. For example, in combination with Fig.10 ,like Fig.11 As shown, after simulating the mouse pressing event and the mouse lifting event, the mobile phone can display a window 1101 on the display screen of the mobile phone, and the window 1101 includes a screenshot of the calculator interface 1001. The mobile phone also performs background blur processing. Fig.11 In the figure, the interface 1001 with a computer as the background is shown as an example.

[0247] The mobile phone (such as the launcher of the mobile phone) can also obtain window information (the window information is the information of the object in this application). The window information may include one or more of the following information: the name of the application in the window, the package name of the application, the class name of the application, the interface screenshot of the application, the size of the window, and the display position of the window on the mobile phone display.

[0248] S908. The PC receives a mouse movement event and a mouse press event, intercepts the mouse movement event and the mouse press event, and sends a mouse operation parameter 4 of the mouse movement event and a mouse operation parameter 5 of the mouse press event to the mobile phone.

[0249] S909. The mobile phone receives mouse operation parameter 4 and mouse operation parameter 5, simulates a mouse move event according to mouse operation parameter 4, simulates a mouse press event according to mouse operation parameter 5, and displays an animation of a window following the movement of the mouse pointer on the display screen of the mobile phone according to the mouse move event and the mouse press event.

[0250] Among them, the mouse operation parameter 4 may be the first operation parameter in the present application, and the mouse operation parameter 5 may be the second operation parameter in the present application.

[0251] When the user wants to drag the interface of the application, the user can move the mouse to trigger the mouse pointer to move from the status bar of the mobile phone display to the window position of S907. After that, the user can long press the mouse and move the mouse at the same time. At this time, the keyboard and mouse module of the PC can receive the press event (such as the mouse press event) and the move event (such as the mouse move event). Similarly, the keyboard and mouse module of the PC will intercept it, and the corresponding operation parameters (such as mouse operation parameters 4 and mouse operation parameters 5) will be captured and sent to the mobile phone through the transmission management module of the PC through the established connection.

[0252] The transmission management module of the mobile phone can receive mouse operation parameters 4 and mouse operation parameters 5, and send them to the drag management module of the mobile phone. After the drag management module of the mobile phone receives mouse operation parameters 4 and mouse operation parameters 5, it converts the key code of the received mouse operation parameter 4 into a key code that the mobile phone can recognize according to the preset mapping relationship, and converts the key code of the received mouse operation parameter 5 into a key code that the mobile phone can recognize. Then, the created virtual input device can be used to simulate a press event, such as a mouse press event, according to the mouse operation parameter 4 after the key code is converted, and the created virtual input device can be used to simulate a move event, such as a mouse move event, according to the mouse operation parameter 5 after the key code is converted. Afterwards, in response, the drag management module of the mobile phone can initiate dragging of the window, and can also draw an animation of the window following the movement of the mouse pointer, and let the mobile phone's launcher display it on the mobile phone's display. For example, combined with Fig.11The user moves the mouse to trigger the mouse pointer to move from the status bar of the mobile phone display to Fig.11 The position of the window in the Fig.12 As shown, the user can long-press the mouse 1201 and move the mouse 1201 at the same time. As the mouse 1201 moves, the mobile phone displays an animation of a window 1202 (the window 1202 includes a screenshot of the calculator interface) following the movement of the mouse pointer 1203 on the display screen of the mobile phone, such as the track of the window 1202 following the movement of the mouse pointer 1203 as shown in track 1204.

[0253] In the process of the window following the movement of the mouse pointer, the mobile phone can determine whether the dragged window is dragged out of the edge of the mobile phone display screen. In this embodiment, the window being dragged out of the edge of the mobile phone display screen may be that a part of the window is dragged out (or overflows) the mobile phone display screen (that is, the proportion of the window overflowing the display screen area is greater than a preset threshold), or the entire area of ​​the window is dragged out (or overflows) the mobile phone display screen, or when the window moves with the mouse pointer on the mobile phone display screen, the mouse pointer slides out of the edge of the mobile phone display screen, and this embodiment does not make specific restrictions here. The following is an example of determining whether the dragged window is dragged out of the edge of the mobile phone display screen, specifically determining whether the mouse pointer slides out of the edge of the mobile phone display screen.

[0254] S910. The mobile phone monitors the coordinate position of the mouse pointer on the mobile phone display screen.

[0255] S911. The mobile phone sends shuttle status information indicating the end of mouse shuttle to the PC when determining that the mouse pointer slides out of the edge of the mobile phone display screen according to the coordinate position of the mouse pointer on the mobile phone display screen, and sends a screenshot of the application interface to the PC.

[0256] Before sending the application interface screenshot to the PC, the mobile phone may also perform serialization processing on it, that is, the application interface screenshot sent by the mobile phone to the PC may be data obtained after serialization processing. The shuttle state information in S911 may be the second shuttle state information in this application.

[0257] S912: The PC receives a mouse movement event, and displays an animation of the application interface screenshot following the movement of the mouse pointer on the display screen of the PC according to the mouse movement event and the received application interface screenshot.

[0258] For example, the user continues to press the mouse and moves the mouse. The PC may receive a movement event, such as a mouse movement event. The PC may draw an application interface screenshot based on the mouse movement event and the received application interface screenshot, and display the screenshot in the form of a window. The PC may also draw an animation of the application interface screenshot moving with the mouse pointer, and display the animation of the application interface screenshot moving with the mouse pointer on the display screen of the PC. For example, in combination with Fig.12 ,like Fig.13 As shown, as the mouse 1301 moves, the PC displays an animation of the application interface screenshot 1302 on the PC display screen following the movement of the mouse pointer 1303, such as the trajectory of the application interface screenshot 1302 following the movement of the mouse pointer 1303 as shown in trajectory 1304. In addition, after the mouse pointer slides over the edge of the mobile phone display screen, the dragging of the application interface on the mobile phone display screen ends, and the mobile phone can display a drag ending animation on the mobile phone display screen. For example, the mobile phone displays an animation of restoring the application interface (or displays an animation of restoring the desktop), such as Fig.13 As shown, after the dragging on the mobile phone is completed, the mobile phone displays the calculator interface 1305.

[0259] That is to say, combined Fig.12 and Fig.13 As shown, by long pressing the mouse and moving the mouse, the user can visually drag the interface of the application (such as the calculator interface) displayed on the mobile phone display from the mobile phone display to the PC display. Of course, if the mobile phone also performs serialization processing on the application interface screenshot before sending it to the PC, the PC can obtain the interface screenshot of the application by deserializing it after receiving the corresponding data.

[0260] In addition, in S911, the mobile phone may also send the size of the window obtained in S907 and the display position of the window on the mobile phone display screen to the PC. Based on these data, the PC may determine the size and initial position of the window used to display the interface screenshot of the application using preset rules. As an example, the preset rule may be that the size of the window is the same as the size of the window obtained in S907, and the initial position of the window is the same as the display position of the window obtained in S907 on the mobile phone display screen. Alternatively, the size of the window is twice the size of the window obtained in S907, and the initial position of the window is the same as the display position of the window obtained in S907 on the mobile phone display screen, etc. The embodiment of the present application does not specifically limit the rule.

[0261] S913: The PC receives a mouse up event, and displays the interface of the application on the display screen of the PC according to the mouse up event.

[0262] After the user drags the interface of the application that he wants to drag (such as the interface of a calculator) to the display screen of the PC, the user can lift his finger (i.e., stop the long press operation on the mouse) and stop moving the mouse. The PC can receive a lift event, such as a mouse lift event. In response to the mouse lift event, the PC can display the interface of the application at the position where the mouse pointer stops moving on the display screen of the PC. For example, in combination with Fig.13 For example, Fig.14 As shown, after the user drags the screenshot of the calculator interface to the display screen of the PC, the user lifts his finger and stops moving the mouse. At this moment, in response, the PC can display the interface 1401 of the calculator on the PC display screen.

[0263] As an example, the specific implementation of the PC displaying the interface of the application on the PC's display screen according to the mouse lift event can be: after the PC's keyboard and mouse module receives the mouse lift event, the PC's keyboard and mouse module can send a request message to the mobile phone through the PC's transmission management module and the connection established with the mobile phone, and the request message is used to request the application interface. After the mobile phone receives the request message, it can obtain the interface of the application according to the application name, package name and class name in the window information obtained in S907, and send it to the PC through the connection established with the PC, so that the PC can display the interface of the application on the PC's display screen. For example, the PC and the mobile phone can use Miracast to display the application interface on the PC display screen. For another example, the mobile phone can use DMP technology to display the application interface on the PC display screen.

[0264] It should be noted that the specific descriptions of other contents in each step of this embodiment are similar to those in Figure 3 The descriptions of the corresponding contents in the illustrated embodiments are similar and will not be repeated here.

[0265] The method provided in this embodiment allows, in a scenario where multiple terminals are used in collaboration, when the same input device, such as a mouse, is used to control all terminals. The user can drag the interface of an application displayed on a terminal from the terminal to other terminals by moving the mouse, thereby increasing the interaction between the terminals when multiple terminals are used in collaboration and improving the user experience.

[0266] It is understandable that, at present, when multi-screen collaboration is used to realize multi-terminal, such as PC and mobile phone collaborative office, the premise of realizing content dragging is to project the interface of the mobile phone to the display screen of the PC for display, and the dragging and processing of the content are all completed using the hardware of the PC. The mobile phone is usually off-screen, and its hardware capabilities (such as display screen, stylus, etc.) cannot participate in collaborative office. This will make it difficult for users to comprehensively utilize the characteristics of different terminals participating in collaborative office, maximize their strengths and avoid their weaknesses, and maximize work efficiency. And the interface projected from the mobile phone to the PC will also occupy the display space of the PC display screen. Affects the efficiency of collaborative use of multiple terminals. The cross-device object dragging method and device provided in this embodiment, without starting screen projection, with the help of keyboard and mouse sharing technology, allows users to use input devices such as touchpads and mice to transfer content (or objects) such as text or files between multiple terminals participating in collaborative use by dragging, and allows users to use these terminals to process the transferred content.

[0267] In some other embodiments, the following combination Figure 1 and Figure 2B Taking the first terminal 101 as a PC, the second terminal 102 as a tablet computer, and the input device 102-1 as a mouse as an example, the cross-device object dragging method provided in the embodiment of the present application is described in detail in conjunction with the accompanying drawings.

[0268] Fig.15 The present invention provides a flow chart of a cross-device object dragging method. In this embodiment, a user uses a PC mouse to transfer content (the content is the dragged object) in a PC to a tablet computer by dragging as an example, and the method provided by this embodiment is described in detail. Fig.15 As shown, the method may include the following S1501-S1510.

[0269] S1501. The tablet computer establishes a connection with the PC.

[0270] After the tablet computer and the PC are successfully connected, the two can be used together. In order to improve the efficiency of the collaborative use, the user can use a set of input devices, such as a PC mouse, to control both the PC and the tablet computer.

[0271] In addition, since the mouse is an input device of the PC, after other terminals establish a connection with the PC, in general, the user can temporarily control the PC using the mouse. In this embodiment, after the PC turns on the keyboard and mouse sharing mode, it can also trigger the other terminals that have established a connection with the PC and created a virtual input device to respond to the mouse operation when it is determined that the mouse shuttle condition is met. Exemplarily, the mouse shuttle condition can be that the mouse pointer displayed on the PC display screen of the mouse corresponds to sliding over the edge of the PC display screen.

[0272] It should be noted that, for the specific description of S1501 of this embodiment, reference may be made to the specific description of the corresponding content in another embodiment S301 of the present application, and will not be repeated here.

[0273] For example, taking the example of a user wanting to transfer the contents of a PC to a tablet computer by dragging and dropping, and continuing to drag on the tablet computer, the process of triggering other terminals connected to the PC that have created a virtual input device, such as a tablet computer, to respond to the mouse operation may include the following S1502-S1506.

[0274] S1502: The PC receives a mouse operation of the selected content.

[0275] The above content can be text (or text), a file, or a folder. The file can include one or more of the following formats, such as word documents, Excel workbooks, PowerPoint presentations, bitmaps, image files, plain text files, sound files, movie files, flash animation files, web page files, compressed files, etc.

[0276] In addition, in this embodiment, the selected content may be one or more. For example, two word documents are selected. Another example is, a word document and an image file are selected. Another example is, two folders are selected.

[0277] S1503: The PC receives a mouse press event and a mouse move event, initiates a drag event according to the mouse press event and the mouse move event, and displays an animation of the shadow of the content moving with the mouse pointer on the display screen of the PC.

[0278] Among them, the mouse pointer of the PC can be the second cursor in this application.

[0279] S1504. The PC monitors the coordinate position of the mouse pointer on the PC display screen.

[0280] S1505. The PC intercepts a mouse movement event based on the coordinate position of the mouse pointer on the PC display screen and determines that the mouse pointer slides out of the edge of the PC display screen, and sends the mouse operation parameters included in the mouse movement event to the tablet computer.

[0281] S1506: The PC obtains a bitmap of the drag event content and the shadow, and sends the bitmap of the drag event content and the shadow to the tablet computer.

[0282] Among them, the drag event content is used to drag the connection end device, such as a tablet computer to initiate a drag event. For example, when the content in S1502 is text, the drag event content may include the text (text). For another example, when the content in S1502 is a file or folder, the drag event content may include a file path (such as a uniform resource identifier (URI)). The drag data in this application may include a bitmap of the drag event content and a shadow, which can be used to drag the connection end device, such as a tablet computer, to display an animation of an object moving with a mouse pointer on its display screen.

[0283] In this embodiment, after the keyboard and mouse sharing mode is turned on, when the user wants to transfer the content of the PC to other terminals connected to the PC and creating virtual input devices, such as a tablet computer, by dragging, the user can use the input device of the PC to select the content to be dragged while continuing to drag in the tablet computer. Subsequently, when the mouse pointer of the PC is displayed on the content, the user can input a drag operation so that the PC can drag the corresponding object, that is, the content (such as the content selected in S1502) according to the drag operation.

[0284] The drag operation may be an operation for indicating the initiation of a drag event for the selected content. The drag operation may include one operation or multiple operations. For example, the drag operation includes two operations, namely a press operation and a move operation. Taking the input device as a mouse as an example, the press operation may be a mouse press operation, and the move operation may be a mouse move operation. The user may press and move the mouse (i.e., use the mouse of the PC to input the mouse press operation and the mouse move operation) to trigger the windows system of the PC to initiate a drag event for the content, so that the content (such as the shadow of the content) may move on the display screen of the PC as the mouse pointer of the PC moves. The shadow may also be referred to as a drag shadow.

[0285] In the process of content (such as the shadow of the content) following the movement of the mouse pointer, the PC can determine whether the dragged content (such as the shadow of the content) is dragged out of the edge of the PC display screen. When the content (such as the shadow of the content) is dragged out of the edge of the PC display screen, the mouse shuttle condition can be triggered. In this embodiment, the content is dragged out of the edge of the PC display screen for a part of the content (such as the shadow of the content) to be dragged out (or overflow) of the PC display screen (that is, the proportion of the window overflowing the display screen area is greater than a preset threshold), or it can be the entire area of ​​the content (such as the shadow of the content) is dragged out (or overflow) of the PC display screen, or it can be that when the content (such as the shadow of the content) moves with the mouse pointer on the PC display screen, the mouse pointer slides out of the edge of the PC display screen, and this embodiment does not make specific restrictions here. The following is an example of determining whether the dragged content (such as the shadow of the content) is dragged out of the edge of the PC display screen, specifically determining whether the mouse pointer slides out of the edge of the PC display screen.

[0286] As an example, the user may continuously move the mouse in the same direction to cause the corresponding mouse pointer displayed on the PC display screen to slide over (or slide out of) the edge of the PC display screen, thereby triggering the mouse shuttling condition.

[0287] Exemplarily, the PC may determine the coordinate position of the mouse pointer on the PC display screen according to the initial position and relative displacement of the mouse pointer, thereby determining whether the mouse pointer slides out of the edge of the PC display screen.

[0288] The initial position of the mouse pointer may be the coordinate position of the mouse pointer on the PC display when the mouse starts to move, or the coordinate position of the mouse pointer on the PC display before the mouse starts to move. The initial position of the mouse pointer may specifically refer to the coordinate position in a coordinate system with the upper left corner of the PC display as the origin, the X axis pointing from the upper left corner to the right edge of the PC display, and the Y axis pointing from the upper left corner to the lower edge of the PC display. For example, the specific process of the PC determining whether the mouse pointer has slipped out of the edge of the PC display may be: combining Figure 4 , the PC can establish a coordinate origin with the initial coordinate position (such as Figure 4 The coordinate system is a coordinate system in which the X axis points from the coordinate origin o to the right edge of the PC display screen, and the Y axis points from the coordinate origin o to the upper edge of the PC display screen. The PC can determine the coordinate values ​​of each edge of the PC display screen in this coordinate system. The coordinate values ​​of each edge of the PC display screen in this coordinate system can be determined based on the resolution of the PC display screen and the initial position of the mouse pointer. For example, Figure 4As shown, in this coordinate system, the coordinate value of the right edge of the PC display screen on the X axis is x1, the coordinate value of the left edge on the X axis is -x2, the coordinate value of the upper edge on the Y axis is y1, and the coordinate value of the lower edge on the Y axis is -y2. After the mouse moves, the mouse will report the relative displacement of the corresponding mouse pointer to the PC. The PC can calculate the coordinate position (x, y) of the mouse pointer on the PC display screen after the mouse moves based on the relative displacement reported by the mouse. Based on the coordinate position (x, y), the PC can determine whether the mouse pointer slides out of the edge of the PC display screen. For example, if the coordinate value x of the mouse pointer on the X axis is greater than x1, it can be determined that the mouse pointer slides out of the right edge of the PC display screen. If the coordinate value x of the mouse pointer on the X axis is less than -x2, it can be determined that the mouse pointer slides out of the left edge of the PC display screen. If the coordinate value y of the mouse pointer on the Y axis is greater than y1, it can be determined that the mouse pointer slides out of the upper edge of the PC display screen. If the coordinate value y of the mouse pointer on the Y axis is less than -y2, it can be determined that the mouse pointer slides out of the lower edge of the PC display screen.

[0289] After determining that the mouse pointer slides out of the edge of the PC display screen, that is, the mouse shuttle condition is triggered, the user can use the PC's input device to control other terminals connected to the PC that have created virtual input devices. In other words, after the mouse shuttle condition is triggered, the PC can send the data of the operation input by the user using the PC's input device to other terminals that have created virtual input devices. For example, if the user continues to move the mouse in the same direction, the PC can intercept the received movement event, such as the mouse movement event, and transmit the operation parameters contained in the mouse movement event, such as the mouse operation parameters, to other terminals connected to the PC that have created virtual input devices, so that the terminal can realize the continuation of the drag event.

[0290] It should be noted that, in this embodiment, if there is only one device connected to the PC, such as the tablet computer mentioned above, and the tablet computer creates a virtual input device, the PC can transmit corresponding operation parameters to the tablet computer so that the tablet computer can implement the continuation of the drag event.

[0291] If there are multiple devices connected to the PC, and some or all of these multiple devices have established virtual input devices, then in some embodiments, the PC may display a list of options on the display screen of the PC when determining that the mouse shuttle condition is triggered, and the list of options includes the identifier of the device that has created the virtual input device among the devices connected to the PC (such as the identifier of the above-mentioned tablet computer). The PC may determine the device that implements the continuation of the drag event based on the user's selection. If the user selects the identifier of the above-mentioned tablet computer, the PC may send the corresponding operation parameters to the tablet computer so that the tablet computer can implement the continuation of the drag event. Among them, after completing the creation of the virtual input device, the device connected to the PC may send an indication message of successful creation of the virtual input device to the PC. The PC may obtain which devices among the devices connected to the PC have successfully created the virtual input device based on the received indication message, and display the above-mentioned list of options based on this. In some other embodiments, the shuttle relationship may be pre-configured. If there are multiple devices connected to the PC, and some or all of these multiple devices have established virtual input devices, the device that implements the drag continuation may be determined based on the pre-configured shuttle relationship. For example, the multiple devices connected to the PC include the tablet computer, and the tablet computer creates a virtual input device. The pre-configured shuttle relationship is that the mouse pointer slides out from the left side (or the left edge) of the PC display screen, then it is determined that the device that implements the drag-and-drop connection is the tablet computer. Then, when the user presses and moves the mouse so that the mouse pointer slides over the left edge of the PC display screen, the PC can not only determine that the mouse shuttle starts, but also determine that the device that implements the drag-and-drop connection is the tablet computer. Of course, if there is one device connected to the PC, and the device creates a virtual input device, it can also be determined whether the device that implements the drag-and-drop connection is the device based on the pre-configured shuttle relationship. For example, if the pre-configured shuttle relationship is that the mouse pointer slides out from the left edge of the PC display screen, the mouse shuttles to the tablet computer. But after the user presses and moves the mouse, the mouse pointer slides over the right edge of the PC display screen, then it can be determined that the mouse does not shuttle to the tablet computer. In some other embodiments, the device that implements the drag-and-drop connection can be determined by identifying the device position. For example, when the user presses and moves the mouse so that the mouse pointer slides over the left edge of the PC display, Bluetooth, ultra-wideband (UWB), ultrasound and other positioning technologies can be used to identify the locations of devices around the PC. For example, if the PC recognizes that the left side of the PC is a tablet computer, it can be determined that the device that implements the drag connection is a tablet computer.

[0292] Among them, the shuttle relationship can be configured in advance by the user through a configuration file, or a configuration interface for configuring the shuttle relationship can be provided to the user, and the user can configure the shuttle relationship in advance through the configuration interface. For example, take the example of a user configuring the shuttle relationship of a tablet terminal through an interface. The PC receives the user's operation to open the configuration interface and displays the configuration interface, which includes the PC's logo (such as the PC icon) and the tablet computer's logo (such as the tablet computer icon), and the user can configure the shuttle relationship by dragging these two logos. As an example, if the user places the tablet computer's logo to the left of the PC's logo, the PC can determine that when the mouse pointer slides over the left edge of the PC display, the device that implements the drag-and-drop connection is the tablet computer. If the user places the tablet computer's logo to the right of the PC's logo, the PC can determine that when the mouse pointer slides over the right edge of the PC display, the device that implements the drag-and-drop connection is the tablet computer. When there are multiple devices, the shuttle relationship of each device can be configured in a pre-configured manner. The following embodiments are all described by taking the determined device for implementing the drag-and-drop continuation as a tablet computer as an example. It should be noted that for the two implementations of determining the device for dragging and continuing based on the pre-configured shuttle relationship and identifying the device position, the above S1501 can be executed before the mouse shuttle is triggered or after the mouse shuttle is triggered, and this embodiment does not make any specific restrictions here.

[0293] Exemplary, combined Figure 2B , after the user uses the mouse to select the content that he wants to drag, the keyboard and mouse module of the PC can receive the corresponding operation, such as the mouse operation of the user selecting the content. The user can move the mouse so that the mouse pointer of the PC is displayed on the content selected by the user. Afterwards, if the user presses and moves the mouse (that is, uses the PC's mouse to input a press operation (such as a mouse press operation) and a move operation (such as a mouse move operation)), the keyboard and mouse module of the PC can receive a press event (such as a mouse press event) and a move event (such as a mouse move event) accordingly. At this time, since the mouse shuttle condition has not been triggered, the mounted HOOK will not intercept the input event, and the mouse press event and mouse move event will be transmitted to the windows system of the PC. Based on the received mouse press event and mouse move event, the windows system of the PC can initiate a drag event for the content and draw the content, such as an animation of the shadow of the content moving with the mouse pointer (or a drag animation) displayed on the PC's display. For example, if Fig.16 As shown in FIG. 1 , the user wants to drag a picture 1601 of the PC to the tablet computer and continue dragging it in the tablet computer. The user selects the picture 1601 using a mouse 1602, and then presses and moves the mouse 1602. As the mouse 1602 moves, the PC displays an animation of a shadow 1606 of the picture 1601 moving along with the mouse pointer 1604 on the display screen 1603 of the PC, as shown in FIG. Fig.16The dragging track of the shadow 1606 of the picture 1601 shown in FIG1604 along with the movement of the mouse pointer 1604 is shown as track 1605. It should be noted that, in this embodiment, the operation of selecting content, such as the mouse operation of selecting content in S1502, is optional. For example, when the content that the user wants to drag is a file or a folder, the mouse operation of selecting the content may not be performed, but when the mouse pointer is displayed on the file or folder, a pressing operation and a moving operation may be performed to initiate a drag event for the file or folder.

[0294] As described in S1501, after the keyboard and mouse sharing mode is turned on, the PC will turn on input monitoring and mount HOOK. After the drag event is initiated, the mouse pointer will move on the PC display screen, and the keyboard and mouse module of the PC can use input monitoring to monitor the real-time coordinate position of the mouse pointer on the PC display screen. When the keyboard and mouse module of the PC determines that the mouse pointer slides out of the edge of the PC display screen based on the real-time coordinate position of the mouse pointer on the PC display screen, it can be determined that the above-mentioned mouse shuttle condition is met. At this time, the keyboard and mouse module of the PC can determine that the mouse shuttle starts.

[0295] After the keyboard and mouse module of the PC determines that the mouse shuttle has started, the keyboard and mouse module of the PC can send shuttle status information indicating that the mouse has started shuttling to the tablet computer through the transmission management module of the PC and the connection established with the tablet computer (the shuttle status information indicating that the mouse has started shuttling may be the first shuttle status information in this application). After receiving the information, the tablet computer can simulate a mouse pointer and display the mouse pointer on the display screen of the tablet computer (the mouse pointer displayed on the tablet computer may be the first cursor in this application). The keyboard and mouse module of the PC can also hide the mouse pointer displayed on the display screen of the PC. Of course, objects that move with the mouse pointer, such as the shadow of the object, will also be hidden. For example, in combination with Fig.16 As the mouse 1602 moves, after the mouse pointer 1604 slides over the edge of the PC display screen 1603, the PC hides the shadow 1606 of the image 1601 and the mouse pointer 1604 displayed on the PC display screen 1603. In addition, the tablet computer displays the mouse pointer on the tablet computer's display screen, giving the user a visual effect that the mouse pointer shuttles from the PC to the tablet computer.

[0296] After the keyboard and mouse module of the PC determines that the mouse shuttle has started, if the user operates the mouse, the keyboard and mouse module of the PC can use HOOK to intercept the corresponding input events received, such as mouse events, and capture the operation parameters in the intercepted mouse events, such as mouse operation parameters. Among them, the mouse operation parameters may include: mouse button flags (used to indicate what operation the user has performed on the mouse, such as pressing, lifting, moving, or scrolling the mouse), coordinate information (when the user moves the mouse, used to indicate the X-coordinate and Y-coordinate of the mouse movement), scroll wheel information (when the user operates the mouse wheel, used to indicate the X-axis distance and Y-axis distance of the scroll wheel), key position information (used to indicate which of the left, middle, or right buttons of the mouse the user has operated). The keyboard and mouse module of the PC can also transmit the captured operation parameters, such as mouse operation parameters, to the tablet computer through the established connection through the transmission management module of the PC, so that the tablet computer can make a corresponding response. For example, continue to combine Fig.16 In the example shown, after the mouse pointer slides over the edge of the PC display screen, the user continues to move the mouse in the same direction. The keyboard and mouse module of the PC can receive movement events, such as mouse movement events. The keyboard and mouse module of the PC can use HOOK to intercept (or shield) it at this time, so that the mouse movement event will not be sent to the windows system of the PC, so that the PC will not respond to the received mouse movement event. The keyboard and mouse module of the PC can also use HOOK to capture the operation parameters of the intercepted mouse movement event, such as mouse operation parameters, and send the captured mouse operation parameters to the tablet computer through the established connection through the transmission management module of the PC. As an example, when the mouse event is a mouse movement event, the corresponding mouse operation parameters can be: a mouse button flag for indicating that the user has moved the mouse, coordinate information for indicating the X coordinate and Y coordinate of the mouse movement, scroll wheel information (value is empty) and key position information (value is empty).

[0297] In addition, after the mouse shuttle starts, the PC (such as the drag management module of the PC) can identify the current drag state of the PC (i.e., whether dragging is in progress). If the PC is currently dragging, it can initiate the continuation of the drag event, or initiate cross-screen dragging. For the windows side, since the drag event needs to be initiated from the window, it also needs to be received by the window. Therefore, after determining that the mouse shuttle has started, the PC can display an invisible window, or a so-called invisible window. Exemplarily, after the keyboard and mouse module of the PC determines that the mouse shuttle has started, the keyboard and mouse module of the PC can send a callback indication of the start of the mouse shuttle to the drag management module of the PC. The drag management module of the PC can send a request for instructing the creation of an invisible window to the window management module of the PC according to the callback indication. The window management module of the PC can create and display an invisible window according to the request. For example, the window management module of the PC can display the invisible window at the edge of the PC display screen. Among them, the transparency of the invisible window is greater than a threshold, such as the transparency of the invisible window is very high, or it is completely transparent.

[0298] After the invisible window is displayed, if the PC is dragging when the mouse shuttle occurs, that is, when the mouse shuttle occurs, the windows system of the PC initiates a drag event for the content, then the invisible window can receive the drag event from the windows system. If the PC does not drag when the mouse shuttle occurs, that is, the user does not select any content when moving the mouse, but only moves the mouse, then the invisible window will not receive the drag event. After the invisible window receives the drag event, the window management module of the PC can obtain the drag event content from the drag event received by the invisible window. For example, the window management module of the PC can capture the drag event content from the drag event through the DragEnter event. After obtaining the drag event content, the window management module of the PC can send it to the tablet computer through the transmission management module of the PC through the connection established with the tablet computer. Among them, before sending the drag event content to the tablet computer, the PC can also serialize it, that is, the drag event content sent by the PC to the tablet computer can be the data obtained after serialization.

[0299] For example, Fig.17As shown, it is a data structure diagram of a drag event on the windows side provided by an embodiment of the present application. When the drag event enters the invisible window, the invisible window will receive the data object corresponding to the drag event, such as IDataObject. The window management module of the PC can attach it to COleDataObject. Afterwards, through the DragEnter event, such as calling the GetData function to obtain the drag event content in the IDataObject corresponding to the drag event. Among them, in this embodiment, the drag event content required to implement the continuation of the drag event may include text or file path, etc. As an example, the window management module of the PC can obtain the text in the IDataObject through the GetData (CF_UNICODETEXT) function. The window management module of the PC can obtain the file path in the IDataObject through the GetData (CF_HDROP) function. After obtaining the text or file path, it can be serialized and sent to the tablet computer.

[0300] In addition, in this embodiment, in order to display the dragged object on the tablet computer, such as the animation of the shadow of the object moving with the mouse pointer, since the shadow and the bitmap can be converted to each other, the PC also needs to obtain the bitmap of the shadow displayed on the PC. For example, the PC can obtain the bitmap of the shadow by intercepting the image of the dragged content displayed on the PC display screen. For another example, taking the dragged content as text as an example, the PC can generate a bitmap of the shadow based on the dragged text. For another example, taking the dragged content as a file as an example, the PC can find the dragged content based on the obtained file path of the dragged content to determine the type of the dragged content (such as an image file), and then use the corresponding default material as the bitmap of the shadow according to the type, or obtain the thumbnail of the content as the bitmap of the shadow according to the obtained file path of the dragged content. The bitmap of the shadow can be sent to the tablet computer through the transmission management module of the PC through the connection established with the tablet computer. Among them, before sending the bitmap of the shadow to the tablet computer, the PC can also perform serialization processing on it, that is, the bitmap of the shadow sent by the PC to the tablet computer can be the data obtained after serialization processing.

[0301] S1507: The tablet computer receives the drag event content and the shadow bitmap, and initiates a drag event according to the drag event content and the shadow bitmap.

[0302] S1508. The tablet computer receives the mouse operation parameters and simulates a mouse movement event according to the mouse operation parameters.

[0303] S1509: The tablet computer generates a mouse press event.

[0304] After the tablet receives the drag event content and the shadow bitmap, it can parse it and initiate a drag event. For the Android system, since the drag event needs to be initiated from the view control, it also needs to be received by the view control. Therefore, the tablet can start a transparent activity, or invisible activity. The invisible activity has a view control with a transparency greater than a threshold. Using the view control, calling the Android open source project (AOSP) interface, the tablet can initiate a corresponding drag event based on the received drag event content and shadow bitmap, so as to achieve the continuation of the drag event on the tablet.

[0305] Exemplary, combined Figure 2B ,and Fig.17 In the example shown, after receiving the bitmap of the drag event content and the shadow, the transmission management module of the tablet computer can transmit the drag event content and the shadow bitmap to the drag management module of the tablet computer. The drag management module of the tablet computer parses the received drag event content to obtain the text or file path from the PC. According to the obtained text or file path, the drag management module of the PC can construct the content data (clipData) of the drag event. In addition, the drag management module of the tablet computer can also generate the corresponding shadow according to the received shadow bitmap. Afterwards, the drag management module of the tablet computer can use the view control of the transparent activity opened by the tablet computer to call the startDragAndDrop method provided by the AOSP interface, and use clipData and shadow as input parameters to initiate the drag event on the tablet computer side. Of course, if the drag event content and the shadow bitmap are serialized by the PC before being sent to the tablet computer, the tablet computer can obtain the drag event content and the shadow bitmap by deserialization after receiving the corresponding data.

[0306] As an example, Fig.18 , which is a data structure diagram of a drag event (DragEvent) on an Android device provided in an embodiment of the present application. Fig.18The drag management module of the tablet computer can construct content data (clipData) according to the text (text) or file path (uri) received from the PC (wherein the text or file path is included in the content (item) included in the content data), and generate the corresponding shadow according to the received shadow bitmap, and then call the startDragAndDrop method of AOSP, and use the clipData and shadow, as well as other parameters obtained according to the user's mouse operation information (such as the received mouse movement event) (such as action (action, which may include start, enter, hover, release, leave, end, etc.), current x coordinate, current y coordinate, local state (localstate), content description (clipdescription), etc.) as input parameters to initiate (dispatch) the drag event (DragEvent) on the tablet computer.

[0307] It should be noted that, combined with Fig.18 , the content description includes a label. The label is used to indicate whether the drag event is initiated by the drag management module of the tablet computer or by the application in the tablet computer. For example, Label is a string. For example, when label is "windowscast", it is used to indicate that the drag event is initiated by the drag management module of the tablet computer; when label is not "windowscast", it is used to indicate that the drag event is not initiated by the drag management module of the tablet computer, but by the application of the tablet computer. In the above example, since the drag event is initiated by the drag management module, the label in the content description of the drag event is "windowscast". The specific description and construction rules of other parameters are similar to the corresponding implementation of generating the original drag event on the Android side in the prior art, and will not be repeated here.

[0308] It is understandable that the execution of the drag event can be triggered by a drag operation, which may include a press operation (such as a mouse press operation) and a move operation (such as a mouse move operation). When the user wants to transfer the content of the PC to the tablet computer by dragging and continuing to drag in the tablet computer, after the user triggers the mouse shuttle by moving the mouse, the user will continue to move the mouse in the same direction. In other words, the user will use the PC input device to input the move operation, such as using the PC mouse to input the mouse move operation. At this time, as described in S1505, after the user uses the PC mouse to input the mouse move operation, the keyboard and mouse module of the PC will intercept the corresponding received move event, such as the mouse move event, and the operation parameters contained in the mouse move event, such as the mouse operation parameter capture and processing, are sent to the tablet computer through the transmission management module of the PC, such as the transmission management module sent to the tablet computer. At this time, the transmission management module of the tablet computer can receive the mouse operation parameter. Since the operating systems of the PC and the tablet computer are different, there are differences in the key values ​​of the mouse operation parameters in their mouse events. Therefore, after receiving the mouse operation parameter, the tablet computer can convert the key code of the received mouse operation parameter into a key code that can be recognized by the tablet computer according to the preset mapping relationship. Afterwards, the tablet computer (such as the keyboard and mouse module of the tablet computer) can simulate input events that can be recognized by the tablet computer, such as mouse events, according to the mouse operation parameter after the key code conversion using the created virtual input device, that is, it can simulate movement events that can be recognized by the tablet computer, such as mouse movement events. The keyboard and mouse module of the tablet computer can also send the mouse movement event to the framework layer of the tablet computer.

[0309] In addition, when the mouse shuttle starts, the shuttle initiator, that is, the PC, is in a dragging state, that is, the PC is in a mouse pressing state, and the shuttle target end, that is, the tablet computer, can only receive the mouse moving event at this time, that is, it is not in the mouse pressing state. Therefore, the tablet computer can generate a pressing operation, such as a mouse pressing operation. For example, after the tablet computer calls the startDragAndDrop method provided by the AOSP interface, the tablet computer can receive a drag start callback (onDragStart). At this time, the tablet computer can determine whether the drag event is initiated by the drag management module of the tablet computer according to the label in the drag event initiated by the tablet computer. When it is determined that the drag event is initiated by the drag management module of the tablet computer (such as label is "windowscast"), the tablet computer can generate (or simulate) a pressing event, such as a mouse pressing event, using the created virtual input device according to the pressing operation, such as the operation parameters of the mouse pressing operation, such as the drag management module of the tablet computer controls the keyboard and mouse module of the tablet computer to send the mouse pressing event to the framework layer of the tablet computer using the created virtual input device. In this way, the drag event initiated in S1507 on the tablet computer side can be attached to the mouse pointer displayed on the display screen of the tablet computer.

[0310] S1510: The tablet computer executes a drag event according to the mouse moving event and the mouse pressing event, and displays an animation of the shadow of the content moving with the mouse pointer on the display screen of the tablet computer.

[0311] The tablet computer (such as the frame layer of the tablet computer) can execute the drag event initiated in S1507 according to a movement event, such as a mouse movement event and a press event, such as a mouse press event. In the process of executing the drag event, the tablet computer can also display content on the display screen of the tablet computer, such as an animation in which the shadow of the content moves with the mouse pointer (the mouse pointer can be the first cursor in this application). For example, in combination with Fig.16 ,like Fig.19 As shown, as the mouse 1901 moves, the tablet computer displays an animation of a shadow 1903 of the picture corresponding to the movement of the mouse pointer 1904 on the display screen 1902 of the tablet computer, such as Fig.19 The dragging trajectory of the shadow 1903 of the picture shown in FIG. 1 as the mouse pointer 1904 moves is shown as trajectory 1905 .

[0312] After the user drags the content from the PC to the tablet computer by dragging, and continues to drag in the tablet computer, since the drag event is attached to the mouse pointer displayed on the tablet computer display screen on the tablet computer side, the user can use the mouse pointer to accurately select the mouse release point. For example, if the user wants to use or process the content in the tablet computer, the user can release the mouse after moving the mouse pointer to the view control of the tablet computer where the user wants to use or process the content. After the mouse is released, the keyboard and mouse module of the PC can receive the corresponding lift event, such as the mouse lift event. Since the mouse lift event is received by the PC after the mouse shuttle starts, the keyboard and mouse module of the PC will use HOOK to intercept it (or shield it) so that the mouse lift event will not be sent to the windows system of the PC, so that the PC will not respond to the received mouse lift event. The keyboard and mouse module of the PC can also use HOOK to capture the operation parameters contained in the mouse lift event, such as the mouse operation parameters, and send the captured mouse operation parameters to the tablet computer through the established connection through the transmission management module of the PC. Among them, the mouse operation parameters of the mouse up event may include: a mouse button flag for indicating that the user has lifted the mouse, coordinate information (the value is empty), scroll wheel information (the value is empty) and key position information for indicating that the user has operated the left button of the mouse. Correspondingly, the transmission management module of the tablet computer can receive the mouse operation parameters. After the tablet computer receives the mouse operation parameters, the tablet computer can convert the key position code of the received mouse operation parameters into a key position code that can be recognized by the tablet computer according to a preset mapping relationship. Afterwards, the tablet computer uses the created virtual input device to simulate the mouse events that can be recognized by the tablet computer according to the operation parameters after the key position code is converted, such as the mouse operation parameters, that is, it can simulate the corresponding input event, such as the mouse up event.

[0313] After simulating the mouse lift event, the tablet computer can determine the mouse release point according to the current coordinate position of the mouse pointer. For example, after the tablet computer learns that the keyboard and mouse shuttle has started, the tablet computer can register a listener for the coordinate position of the mouse pointer. In this way, the tablet computer can monitor the coordinate position of the mouse pointer on the tablet computer display screen in real time through the listener. Based on this, after the tablet computer receives the mouse lift event, the tablet computer can obtain the current coordinate position of the mouse pointer on the tablet computer display screen using the listener. According to the obtained coordinate position, the tablet can determine the mouse release point. For another example, after the tablet computer calls the startDragAndDrop method of AOSP to initiate a drag event, the tablet computer will listen to the input event. For example, when the user continues to drag in the tablet computer, the tablet computer can listen to a moving event, such as a mouse moving event, and according to the mouse moving event, the tablet computer can obtain the operation parameters of the mouse moving event, such as the mouse operation parameters, such as extracting the parameters from the MotionEvent. The mouse operation parameters include coordinate information indicating the position of the mouse. Afterwards, when the user lifts his finger, the tablet computer monitors a lift event, such as a mouse lift event, and can determine the coordinate position of the mouse pointer based on previously obtained coordinate information, thereby determining the mouse release point based on the obtained coordinate position.

[0314] After the user releases the mouse, since the PC directly intercepts and processes the received lift event, such as the mouse lift event, the drag event on the PC is not released. At this time, after receiving the mouse lift event, the tablet computer can send a signal to the PC through the established connection to inform the PC to release the drag event.

[0315] In addition, the tablet computer can also respond to the lift event, such as the mouse lift event. For example, taking the content in S1502 as text as an example, after the tablet computer simulates the mouse lift event, the drag management module of the tablet computer can send the content data in the above drag event to the view control at the mouse release point, and the content data includes the text. After the view control receives the content data, it can perform corresponding processing according to the text in the content data, such as displaying the text in the view control. For another example, the content in S1502 is a file as an example. Since the file is actually not transferred to the tablet computer during the dragging process of the tablet computer. Therefore, after the user releases the mouse and the PC receives the above-mentioned signaling from the tablet computer notifying the release of the drag event, the PC can transfer the file to the tablet computer. After the tablet computer receives the file, it can store the file in a predetermined cache directory. At this time, the drag management module of the tablet computer can obtain the uri of the file (such as uri 1), which is the path of the file in the cache directory, which is different from the uri in the drag event content sent by the PC to the tablet computer in S1507 (the uri sent by the PC is the storage path of the file on the PC). The drag management module of the tablet computer can construct new content data according to URI 1, and send the content data to the view control at the mouse release point as a response to the mouse lift event. After receiving the content data, the view control can perform corresponding processing. For example, if the view control is a view control in a memo, the file can be displayed. If the view control is an input box in a chat window, the file can be sent out.

[0316] In some other embodiments, when the tablet computer is also connected to other devices, such as a mobile phone, if the user wants to continue dragging the content to the mobile phone, the user can continue to move the mouse so that the mouse pointer on the tablet computer slides over the edge of the tablet computer display screen, triggering the mouse to shuttle from the tablet computer to the mobile phone, so as to continue to implement the continuation of the drag event on the mobile phone. It should be noted that after the mouse shuttles to the mobile phone, the PC can directly interact with the mobile phone so that the mobile phone can implement the continuation of the drag event. The specific description of implementing the continuation of the drag event on the mobile phone is similar to the specific description of implementing the continuation of the drag event on the tablet computer, and will not be repeated here.

[0317] It is understandable that through the above process, without starting screen projection, the user can use an input device, such as a PC mouse, to drag the text or file content in the PC application to the edge of the PC screen and continue to move in the same direction to trigger mouse shuttle. After the mouse shuttle starts, the mouse pointer appears on the tablet computer. And the PC sends the drag event content to the tablet computer, so that the drag event can be continued to be attached to the mouse pointer of the tablet computer to realize the continuation of the drag event on the tablet computer, giving the user a visual effect of content being dragged from the PC to the tablet computer.

[0318] In an embodiment of the present application, the user may not only have the need to transfer the content of the PC to the tablet computer by dragging and continuing to drag in the tablet computer, but may also have the need to transfer the content in the tablet computer to the PC by dragging. For example, in combination with the examples in S1501-S1510 above, the user transfers the picture 1 of the PC to the tablet computer by dragging, and releases the mouse after continuing to drag in the tablet computer. Afterwards, the user edits the picture 1 with the stylus of the tablet computer and saves it. The user wants to transfer the edited picture 1 to the PC by dragging, and releases the mouse after continuing to drag on the PC, so as to save the edited picture 1 on the PC.

[0319] Please refer to Fig. 20 , is a flowchart of another cross-device object dragging method provided by an embodiment of the present application. Fig.15 In the embodiment shown, the method provided by this embodiment is introduced by taking the case where the user wants to transfer the content of the tablet computer to the PC by dragging after the mouse shuttle occurs, and continues to drag in the PC as an example. Fig. 20 As shown, the method may include the following S2001-S2010.

[0320] S2001. The tablet computer receives a mouse operation of selected content from the PC.

[0321] S2002. The tablet computer receives mouse operation parameters of mouse press events and mouse move events from the PC, simulates mouse press events and mouse move events according to the mouse operation parameters, initiates a drag event according to the mouse press events and mouse move events, and displays an animation of the shadow of the content moving with the mouse pointer on the display screen of the tablet computer.

[0322] Among them, in this embodiment, the mouse operation parameter of the mouse press event can be the first operation parameter in this application, and the mouse operation parameter of the mouse move event can be the second operation parameter in this application.

[0323] like Fig.15In the description of the illustrated embodiment, after the keyboard and mouse sharing mode is turned on, if mouse shuttle occurs, the PC will use the mounting HOOK to intercept the corresponding input event, such as the mouse event, after the user operates the PC input device, such as the mouse, and the PC will capture the operation parameters contained in the intercepted mouse event, such as the mouse operation parameters, and send them to the tablet computer. For example, after the mouse shuttles to the tablet computer, the user wants to transfer the content in the tablet computer to the PC by dragging, and continues to drag in the PC. Combined with Figure 2B , the user uses the PC's input device, such as a mouse, to select the content that he wants to drag. The PC's keyboard and mouse module can receive corresponding input operations, such as mouse operations. The PC's keyboard and mouse module will use HOOK to intercept the corresponding input events received, such as mouse events, so that the input events will not be sent to the PC's windows system, so that the PC will not respond to the received input events. The PC's keyboard and mouse module will also use HOOK to capture the operation parameters in the intercepted input events, such as mouse operation parameters, and transmit them to the tablet computer through the established connection through the PC's transmission management module. At this time, the tablet computer's transmission management module can receive the corresponding operation parameters, such as mouse operation parameters. After the tablet computer converts the received operation parameters into key codes, it can simulate the corresponding input events, such as mouse events, using the created virtual input device.

[0324] Afterwards, when the mouse pointer of the tablet computer is displayed on the object to be dragged, such as the content selected by the user, the user can use the input device of the PC, such as the mouse, to input the drag operation, so that the tablet computer can realize the dragging of the corresponding object, that is, the content, according to the drag operation. Among them, the drag operation can be an operation for indicating the initiation of a drag event for the selected content. The drag operation can include one operation or multiple operations. For example, the drag operation includes two operations, namely, a press operation and a move operation. Taking the input device as a mouse as an example, the press operation can be a mouse press operation, and the move operation can be a mouse move operation. For example, the user can press and move the mouse of the PC, that is, use the mouse of the PC to input the mouse press operation and the mouse move operation. Afterwards, the keyboard and mouse module of the PC can receive the press event (such as the mouse press event) and the move event (such as the mouse move event) correspondingly. Similarly, the keyboard and mouse module of the PC can use HOOK to intercept the received mouse press event and mouse move event, and use HOOK to capture the operation parameters of the intercepted mouse press event and mouse move event, such as the mouse operation parameters, and transmit them to the tablet computer through the established connection through the transmission management module of the PC. At this time, the transmission management module of the tablet computer can receive the corresponding operation parameters, such as mouse operation parameters. After converting the received operation parameters into key codes, the tablet computer can simulate corresponding input events using the created virtual input device, such as simulating press events (such as mouse press events) and move events (such as mouse move events).

[0325] After the tablet simulates a press event (such as a mouse press event) and a move event (such as a mouse move event), in response to these events, the tablet can initiate a drag event from the corresponding application (such as if the selected content is a word document, the application is an office application; another example is if the selected content is a picture in a file manager, the application is a file manager; another example is if the selected content is a text in a memo, the application is a memo) and draw the content, such as the shadow of the content is displayed on the tablet's display screen as the mouse pointer moves. For example, a user uses the tablet's stylus to edit and save Picture 1 dragged from the PC. The user wants to drag the edited picture to the PC and continue dragging it in the PC. Fig.21 As shown, the user can use the mouse 2101 to select the edited picture 2102, and then press and move the mouse 2101. As the mouse 2101 moves, the tablet computer displays an animation of the shadow 2106 of the edited picture 2102 moving with the mouse pointer 2104 on the display screen 2103 of the tablet computer, as shown in FIG. Fig.21 The dragging trajectory of the shadow 2106 of the picture 2102 shown in the figure as the mouse pointer 2104 moves is shown as trajectory 2105.

[0326] It should be noted that the above description is based on the example that the event triggering dragging includes a press event and a move event. For example, the user can trigger dragging by pressing and moving the mouse. In some other embodiments, the event triggering dragging may only include a press event. For example, the user can trigger dragging by long pressing the mouse, or the user can trigger dragging by pressing the mouse. This embodiment is not specifically limited here.

[0327] S2003: The tablet computer obtains a bitmap of the drag event content and the shadow, and caches the obtained bitmap of the drag event content and the shadow.

[0328] The drag event content is used to drag the connection end device to construct a drag event. For example, when the content in S2001 is text, the drag event content may include text. For another example, when the content in S2001 is a file or folder, the drag event content may include a file path (such as uri).

[0329] Exemplary, combined Figure 2B , when the application of the tablet computer initiates a drag event, the corresponding drag event content can be called back to the application layer by the framework layer. For example, when the application calls the startDragAndDrop method provided by the AOSP interface to initiate a drag event, the framework layer can use program instrumentation to extract the drag event content from it. For example, the location and content that need to be instrumented can be determined in advance, and then program instrumentation can be performed according to the determined location and content so that the drag event content can be extracted. Afterwards, the framework layer of the tablet computer can call back the extracted drag event content to the drag management module of the tablet computer application layer. After the drag management module of the tablet computer obtains the drag event content, it can cache it. As an example, the drag event content required to implement the continuation of the drag event may include text or file paths, etc. When the application initiates a drag event, the framework layer can extract text or file paths by using program instrumentation.

[0330] In addition, in this embodiment, the drag event may be initiated in two ways. One is initiated by the drag management module of the tablet computer (that is, the drag is from the PC to the tablet computer, such as Fig.15 The other is initiated by an application on a tablet computer. Fig.15 In the embodiment shown, Fig.18As can be seen from the description of the data structure shown, in this embodiment, the content description of the drag event of the tablet computer includes a label, which is used to indicate whether the drag event is initiated by the drag management module of the tablet computer or by the application in the tablet computer. It can be understood that when the drag event is initiated by the application of the tablet computer, or not by the drag management module of the tablet computer, it is necessary to obtain the drag event content and cache it, so that after the drag is shuttled to the PC, it is used for the PC to continue the drag event. That is to say, after the drag event is initiated, the tablet computer can distinguish whether the drag event is initiated by the application of the tablet computer or the drag management module of the tablet computer according to the label in the content description of the drag event. For example, when the label is not "windowscast", it can be determined that the drag event is not initiated by the drag management module of the tablet computer, but by the application of the tablet computer. When the drag event is not initiated by the drag management module of the tablet computer, the framework layer needs to obtain the drag event content and send the obtained drag event content to the drag management module of the tablet computer so that it can cache the drag event content.

[0331] In addition, in this embodiment, the framework layer can also obtain the bitmap of the shadow through a newly added interface or by using an existing interface (such as by using the interface for calling back clipData). The obtained bitmap of the shadow can also be called back to the drag management module of the tablet computer application layer. After the drag management module of the tablet computer obtains the bitmap of the shadow, it can also cache it.

[0332] S2004. The tablet computer monitors the coordinate position of the mouse pointer on the tablet computer display screen.

[0333] S2005. The tablet computer sends shuttle status information indicating the end of mouse shuttle to the PC when determining that the mouse pointer slides out of the edge of the tablet computer display screen based on the coordinate position of the mouse pointer on the tablet computer display screen.

[0334] S2006. The PC receives shuttle status information indicating the end of mouse shuttle.

[0335] The shuttle state information used to indicate the end of mouse shuttle may be the second shuttle state information in the present application.

[0336] In the process that the content (such as the shadow of the content) follows the movement of the mouse pointer, the tablet computer can determine whether the dragged content (such as the shadow of the content) is dragged out of the edge of the tablet computer display screen. When the content (such as the shadow of the content) is dragged out of the edge of the tablet computer display screen, it indicates that the user wants to use the mouse to control other devices. In this embodiment, the specific description of the content being dragged out of the edge of the tablet computer display screen is similar to the specific description of the content being dragged out of the PC, and will not be repeated here. The following is an example of determining whether the dragged content (such as the shadow of the content) is dragged out of the edge of the tablet computer display screen, specifically determining whether the mouse pointer slides out of the edge of the tablet computer display screen.

[0337] As an example, after the drag event is initiated, the mouse pointer will move on the display screen of the tablet computer, and the tablet computer can monitor the real-time coordinate position of the mouse pointer on the display screen of the tablet computer. For example, after the keyboard and mouse shuttle starts, the tablet computer can register a listener for the coordinate position of the mouse pointer. In this way, the tablet computer can monitor the coordinate position of the mouse pointer on the display screen of the tablet computer in real time through the listener. The tablet computer can determine whether the mouse pointer slides over the edge of the display screen of the tablet computer according to the real-time coordinate position of the mouse pointer on the display screen of the tablet computer monitored by the listener. Exemplarily, the tablet computer can determine the coordinate position of the mouse pointer on the display screen of the tablet computer according to the initial position and relative displacement of the mouse pointer, thereby determining whether the mouse pointer slides out of the edge of the display screen of the tablet computer. Among them, the initial position of the mouse pointer can be the coordinate position of the mouse pointer on the display screen of the tablet computer when the mouse starts to move, or the coordinate position of the mouse pointer on the display screen of the tablet computer before the mouse starts to move. The initial position of the mouse pointer can specifically refer to the coordinate position in the coordinate system with the upper left corner of the display screen of the tablet computer as the coordinate origin, the X axis points from the upper left corner to the right edge of the display screen of the tablet computer, and the Y axis points from the upper left corner to the lower edge of the display screen of the tablet computer. The specific implementation of the tablet computer determining that the mouse pointer slides out of the edge of the tablet computer display screen is similar to the specific implementation of the PC determining that the mouse pointer slides out of the edge of the PC display screen, and will not be described in detail here.

[0338] After the tablet computer determines that the mouse pointer slides over the edge of the tablet computer display screen, it indicates that the user wants to use the mouse to control other devices. As described in S1506, if the tablet computer is connected to only one device, the PC, it indicates that the user wants to use the mouse to control the PC. If the tablet computer is connected to multiple devices, the mobile phone can display a list of options, which includes the identifiers of all devices connected to the tablet computer for the user to select the device that he wants to control with the mouse. If the user selects the identifier of the PC, it indicates that the user wants to use the mouse to control the PC. Alternatively, a shuttle relationship can be pre-configured in the tablet computer to determine which device the mouse shuttles to, that is, to determine which device responds to the mouse operation. The specific description of the configuration and application of the shuttle relationship is similar to the description of the corresponding content in the above embodiment, and will not be repeated here. When it is determined that the user wants to use the mouse to control the PC, the tablet computer can determine that the mouse shuttle is over. At this time, the tablet computer can send shuttle status information indicating the end of the mouse shuttle to the PC through the established connection.

[0339] S2007. The PC determines that the mouse shuttle is finished according to the received shuttle status information.

[0340] S2008. The PC cancels the interception of the mouse event and generates a mouse press event.

[0341] S2009. The PC obtains the drag event content and the shadow bitmap from the tablet computer, and initiates a drag event according to the drag event content and the shadow bitmap.

[0342] The drag data in the present application may include the drag event content and a bitmap of a shadow. After the PC receives the shuttle state information indicating the end of the mouse shuttle, it may be determined that the mouse shuttle ends.

[0343] After determining that the mouse shuttle is completed, the PC can display the mouse pointer on the PC display. Fig.15 In the description of the illustrated embodiment, when the mouse shuttle starts, the PC will hide the mouse pointer on the PC display screen. Therefore, after determining that the mouse shuttle is over, the PC can re-display the mouse pointer on the PC display screen. In addition, the trigger condition for the start of the mouse shuttle is that the mouse pointer slides over the edge of the PC display screen. Therefore, the mouse pointer is displayed at the edge of the PC display screen before being hidden. In this embodiment, when the mouse shuttle ends, the PC cancels the hiding of the mouse pointer, and the mouse pointer will be displayed at the edge of the PC display screen. Of course, when the mouse shuttle ends, the mouse pointer on the tablet computer will not be displayed. This gives the user a visual effect of the mouse pointer shuttling from the tablet computer to the PC.

[0344] After confirming that the mouse shuttle has ended, the PC also needs to uninstall HOOK (or close HOOK) to cancel the interception of input devices, such as mouse events. For example, after the user moves the mouse to transfer the content of the tablet to the PC by dragging, triggering the mouse to end the shuttle, the user will continue to move the mouse in the same direction, and the keyboard and mouse module of the PC can receive movement events, such as mouse movement events. Since the HOOK has been uninstalled at this time, the keyboard and mouse module of the PC will send the received movement events, such as mouse movement events, to the windows system of the PC so that the windows system of the PC can respond to the movement events.

[0345] In addition, before the mouse shuttle ends, the tablet computer is in a dragging state, and the mouse shuttles back to the PC to achieve the continuation of the drag, that is, the drag needs to be continued on the PC. Because the Windows platform must press the mouse on the window to initiate the drag normally. However, while the user continues to move the mouse, the PC can only receive movement events, such as mouse movement events, that is, it will not receive press events, such as mouse press events. Therefore, the PC (such as the drag management module of the PC) can generate a press event, such as a mouse press event, and transmit it to the invisible window. For example, after the PC determines that the mouse shuttle is over and the tablet computer is in the process of dragging, the PC can generate a mouse press event and transmit it to the invisible window, so that the drag event initiated in PC-side S2009 can be attached to the mouse pointer displayed on the PC display, that is, the continuation of the drag event is achieved on the PC side.

[0346] In this embodiment, after determining that the mouse shuttle is finished, the PC may also request the tablet computer for the drag state of the tablet computer (i.e., whether it is being dragged). When the drag state returned by the tablet computer indicates that it is being dragged, the PC may request the tablet computer for the drag event content and the bitmap of the shadow. The PC may send a request message to the tablet computer, and the request message may be used to request the drag data, i.e., for requesting the bitmap of the drag event content and the shadow.

[0347] Exemplary, combined Figure 2BWhen the PC determines that the mouse shuttle has ended, the keyboard and mouse module of the PC may send an indication of the end of the mouse shuttle to the drag management module of the PC. The drag management module of the PC may request the drag status from the tablet computer through the transmission management module of the PC based on the received indication. When the drag status returned by the tablet computer indicates that it is being dragged, the drag management module of the PC may request the drag event content and the bitmap of the shadow from the tablet computer through the transmission management module of the PC. Correspondingly, the transmission management module of the tablet computer may receive the request and forward the request to the drag management module of the tablet computer. After receiving the request, the drag management module of the tablet computer may feed back the drag event content and the bitmap of the shadow cached in S2003 to the transmission management module of the PC through the transmission management module of the tablet computer.

[0348] Combined with the description in S2003, after the transmission management module of the PC receives the drag event content, it can transmit the drag event content to the drag management module of the PC. The drag management module of the PC parses the received drag event content and can obtain the text or file path from the tablet computer. Based on the obtained text or file path, the drag management module of the PC can construct a data object of the drag event, such as IDataObject. In addition, after the transmission management module of the PC receives the bitmap of the shadow, it can restore the shadow on the PC side according to the bitmap. For example, the IDragSourceHelper interface provided by the PC side can be used to restore the shadow. After that, the PC can initiate a drag event on the PC side. For example, combined with Fig.15 In the description of the illustrated embodiment, when the mouse shuttle starts, the PC will display an invisible window. Therefore, the PC can use the displayed invisible window to initiate the drag event. After the drag event is initiated, the invisible window can be closed.

[0349] S2010. The PC executes a drag event according to the mouse moving event and the mouse pressing event, and displays an animation of the shadow of the content moving with the mouse pointer on the display screen of the PC.

[0350] After the drag event is initiated, the PC may execute the drag event in response to a move event, such as a mouse move event, and a press event, such as a mouse press event. The PC may also display content on the display screen of the PC, such as an animation of a shadow of the content moving with the mouse pointer. For example, in combination with Fig.21 ,like Fig. 22 As shown, as the mouse 2201 moves, the PC displays an animation of the shadow 2203 of the edited image moving along with the mouse pointer 2204 on the display screen 2202 of the PC, as shown in FIG. Fig. 22 The dragging track of the shadow 2203 shown in FIG. 2 as the mouse pointer 2204 moves is shown as track 2205 .

[0351] It should be noted that the above example is based on the example that the user first drags the content from the PC to the tablet computer, and then drags the content from the tablet computer to the PC. In some other embodiments, the user may not drag the content from the PC to the tablet computer, but directly drag a certain content from the tablet computer to the PC. The specific implementation of this embodiment is different from Fig.15 and Fig. 20 The specific implementation of the illustrated embodiment is similar, except that S1502 is not executed. S1503 is replaced by receiving a moving event (such as a mouse moving event), and displaying an animation of the movement of the mouse pointer on the display screen of the PC according to the moving event. S1506 is not executed, but the operation of the PC displaying an invisible window needs to be executed, but the invisible window will not receive a drag event. S1507 and S1509 are not executed. S1510 is replaced by the tablet computer displaying an animation of the movement of the mouse pointer on the display screen of the tablet computer according to a moving event, such as a mouse moving event. That is, after the mouse shuttle starts, the tablet computer can receive an operation of the user using the input device of the PC, such as a mouse input, to move the mouse pointer (such as called the first cursor) displayed on the tablet computer end. In response to this operation, the tablet computer can display an animation of the movement of the mouse pointer on the display screen of the tablet computer. If the user wants to drag a certain content, he can continue to move the mouse of the PC until the mouse pointer moves to the content. The tablet computer receives an operation of moving the first cursor input by a user using an input device of the PC, such as a mouse. Specifically, the PC intercepts a corresponding movement event, such as a mouse movement event, and sends an operation parameter included in the movement event, such as a third operation parameter, to the tablet computer. The tablet computer can simulate a movement event, such as a mouse movement event, according to the third operation parameter, so that an animation of the movement of the mouse pointer can be displayed on the display screen of the tablet computer. Other operations are similar, and the embodiments of the present application will not be described in detail here.

[0352] The above embodiment is described by taking the example that the tablet computer caches the drag data when the drag starts, and the PC requests the drag data from the tablet computer after the mouse shuttle ends. In some other embodiments, the tablet computer may not cache the drag data when the drag starts, but obtains and actively sends the drag data to the PC after determining that the mouse shuttle ends, without the PC requesting.

[0353] The method provided in this embodiment, without starting screen projection, uses keyboard and mouse sharing technology to allow users to use input devices such as a mouse to drag and drop text, files and other content to follow the mouse pointer between multiple terminals participating in collaborative use. And allowing users to use these terminals to process the transmitted content, so that the hardware capabilities of these multiple terminals can all participate in collaborative office. In addition, since there is no need to start screen projection, it will not occupy the display space of a terminal display. The efficiency of collaborative use of multiple terminals is improved, and the user experience is improved.

[0354] In addition, the above embodiment is described by taking the input device as a mouse as an example. In this embodiment, the input device may also be a touch pad. When the input device is a touch pad, the user can use the button (left button or right button) of the touch pad to input a press operation, and slide the finger on the touch pad to input a move operation. The specific implementation of the user using the touch pad to drag an object is similar to the specific implementation of using a mouse to drag an object, and will not be repeated here.

[0355] Fig.23 A schematic diagram of the composition of a cross-device object dragging device provided in an embodiment of the present application. Fig.23 As shown, the device can be applied to a second terminal (such as the above-mentioned mobile phone or tablet computer), the second terminal is connected to the first terminal, and the device may include: a display unit 2301, a receiving unit 2302 and a sending unit 2303.

[0356] The display unit 2301 is used to display a first cursor on the object displayed on the second terminal.

[0357] The receiving unit 2302 is configured to receive a drag operation input by a user using an input device of the first terminal, where the drag operation is used to initiate dragging of an object.

[0358] The display unit 2301 is further configured to display an animation of the object moving along with the first cursor on the display screen of the second terminal in response to the drag operation.

[0359] The sending unit 2303 is configured to send the drag data to the first terminal when it is determined that the object is dragged out of the edge of the display screen of the second terminal.

[0360] Furthermore, the drag data may be used by the first terminal to display an animation of the object moving along with the second cursor on the display screen of the first terminal.

[0361] Further, the object is an application icon; or, the object is a window, and the window includes an application interface; when the object is an application icon, the drag data includes: the application icon; when the object is a window, the drag data includes: the application interface.

[0362] Furthermore, the receiving unit 2302 is further configured to receive an indication message from the first terminal, where the indication message is sent by the first terminal after the user uses an input device of the first terminal to input a drag-and-release operation, and is configured to indicate that the dragging of the object is ended.

[0363] The sending unit 2303 is further configured to send data to the first terminal, where the data is used by the first terminal to display an interface of the application on a display screen of the first terminal.

[0364] Furthermore, when the object is an icon of an application, the device may further include: a creating unit 2304 .

[0365] The creating unit 2304 is configured to create a drag object according to the icon of the application, where the drag object includes: an icon that is the same as the icon of the application.

[0366] The display unit 2301 is specifically configured to display an animation of the dragged object moving along with the first cursor in response to the drag operation.

[0367] Furthermore, when the object is a window, the display unit 2301 is also used to display the interface of the application; the receiving unit 2302 is also used to receive a long press operation input by the user using the input device of the first terminal when the first cursor is displayed on the status bar of the second terminal, so as to trigger the windowing of the interface of the application; the display unit 2301 is also used to display the window in response to the long press operation; the window specifically includes a screenshot of the interface of the application, and the drag data specifically includes a screenshot of the interface of the application.

[0368] Furthermore, the display unit 2301 is specifically configured to display an animation of the shadow of the object moving along with the first cursor on the display screen of the second terminal.

[0369] Furthermore, the object is text, a file or a folder; the drag data includes the drag event content and a shadow bitmap; wherein, when the object is text, the drag event content includes the text, and when the object is a file or a folder, the drag event content is the file path.

[0370] Furthermore, the device may also include: an acquisition unit 2305, configured to acquire and cache the drag data.

[0371] Furthermore, the receiving unit 2302 is further configured to receive a request message from the first terminal, where the request message is used to request the drag data; and the sending unit 2303 is specifically configured to send the cached drag data to the first terminal.

[0372] Furthermore, the receiving unit 2302 is further configured to receive first shuttling state information from the first terminal, where the first shuttling state information is used to indicate the start of shuttling.

[0373] The display unit 2301 is further configured to display a first cursor on the second terminal in response to the first shuttle state information.

[0374] Furthermore, the receiving unit 2302 is further configured to receive an operation of moving the first cursor input by the user using an input device of the first terminal.

[0375] The display unit 2301 is further configured to display an animation of the movement of the first cursor on the display screen of the second terminal in response to the operation of moving the first cursor, until the first cursor moves onto the object.

[0376] Furthermore, the drag operation includes a pressing operation and a moving operation; the receiving unit 2302 is specifically used to receive a first operation parameter from the first terminal, the first operation parameter is the operation parameter included in the pressing event received by the first terminal after the user uses the input device of the first terminal to perform a pressing operation; and receive a second operation parameter from the first terminal, the second operation parameter is the operation parameter included in the moving event received by the first terminal after the user uses the input device of the first terminal to perform a moving operation.

[0377] The device may further include: a simulation unit 2306, configured to simulate a press event according to the first operation parameter, and simulate a move event according to the second operation parameter, wherein the press event and the move event are used to indicate the initiation of dragging the object.

[0378] The display unit 2301 is specifically configured to display an animation of an object moving along with a first cursor in response to a pressing event and a moving event.

[0379] Furthermore, the receiving unit 2302 is specifically configured to receive a third operation parameter from the first terminal, where the third operation parameter is an operation parameter included in a movement event received by the first terminal after the user uses an input device of the first terminal to perform a movement operation.

[0380] The simulation unit 2306 is used to simulate a moving event according to the third operation parameter, where the moving event is used to indicate moving the first cursor.

[0381] Furthermore, the sending unit 2303 is further configured to send second shuttling status information to the first terminal, where the second shuttling status information is used to indicate the end of the shuttling.

[0382] Furthermore, the long press operation includes a pressing operation and a lifting operation; the receiving unit 2302 is specifically used to receive a fourth operation parameter from the first terminal, the fourth operation parameter is the operation parameter contained in the pressing event received by the first terminal after the user uses the input device of the first terminal to perform a pressing operation when the first cursor is displayed on the status bar; receive a fifth operation parameter from the first terminal, the fifth operation parameter is the operation parameter contained in the lifting event received by the first terminal after the user performs the pressing operation for a period of time and uses the input device of the first terminal to perform a lifting operation.

[0383] The simulation unit 2305 is used to simulate a press event according to the fourth operation parameter and to simulate a lift event according to the fifth operation parameter. The press event and the lift event are used to trigger the windowing of the application interface.

[0384] The display unit 2301 is specifically configured to display a window in response to a press event and a lift event.

[0385] Furthermore, the creation unit 2304 is also used to create a virtual input device after the connection with the first terminal is successfully established; or, the receiving unit 2302 is also used to receive a notification message from the first terminal, the notification message is used to indicate that the keyboard and mouse sharing mode of the first terminal has been turned on, and the creation unit 2304 is also used to create a virtual input device in response to the notification message; wherein the virtual input device is used for the second terminal to simulate input events according to operating parameters.

[0386] Fig.24 A schematic diagram of another cross-device object dragging device provided in an embodiment of the present application. Fig.24 As shown, the device can be applied to a first terminal (such as the above-mentioned PC), the first terminal is connected to a second terminal, and the device can include: a sending unit 2401 and a receiving unit 2402.

[0387] The sending unit 2401 is configured to send data of a drag operation input by a user using an input device of the first terminal to the second terminal, where the drag operation is used to initiate dragging of an object displayed on the second terminal.

[0388] The receiving unit 2402 is used to receive drag data from the second terminal. The drag data is sent to the first terminal after the second terminal determines that the object is dragged out of the edge of the display screen of the second terminal when the object moves with the first cursor on the display screen of the second terminal.

[0389] Furthermore, the device may also include: a display unit 2403, configured to display an animation of the object moving along with the second cursor on the display screen of the first terminal according to the drag data and the drag operation.

[0390] Further, the object is an application icon; or, the object is a window, and the window includes an application interface; when the object is an application icon, the drag data includes: the application icon; when the object is a window, the drag data includes: the application interface.

[0391] Furthermore, the receiving unit 2402 is further configured to receive a drag-and-release operation input by the user using an input device of the first terminal.

[0392] The display unit 2403 is further configured to display the interface of the application on the display screen of the first terminal in response to a drag-and-release operation.

[0393] Furthermore, the sending unit 2401 is further configured to send an indication message to the second terminal, where the indication message is used to indicate that the dragging of the object is finished.

[0394] The receiving unit 2402 is further configured to receive data from the second terminal.

[0395] The display unit 2403 is specifically configured to display the interface of the application on the display screen of the first terminal according to the data.

[0396] Furthermore, when the object is a window, the sending unit 2401 is also used to send data of a long press operation input by the user using the input device of the first terminal to the second terminal, where the long press operation is input by the user when the first cursor is displayed on the status bar of the second terminal, and is used to trigger the windowing of the interface of the application displayed on the second terminal; the window specifically includes a screenshot of the interface of the application, and the drag data specifically includes a screenshot of the interface of the application.

[0397] Furthermore, the display unit 2403 is specifically configured to display an animation of the shadow of the object moving along with the second cursor on the display screen of the first terminal according to the drag data and the drag operation.

[0398] Furthermore, the object is text, a file or a folder; the drag data includes the drag event content and a shadow bitmap; wherein, when the object is text, the drag event content includes the text, and when the object is a file or a folder, the drag event content is the file path.

[0399] Furthermore, the device may also include: a determining unit 2404, configured to determine that the second cursor slides out of the edge of the display screen of the first terminal.

[0400] Furthermore, the receiving unit 2402 is further configured to receive an operation of moving the second cursor input by the user using an input device of the first terminal.

[0401] The display unit 2403 is further configured to display an animation of the movement of the second cursor on the display screen of the first terminal in response to an operation of moving the second cursor.

[0402] Furthermore, the sending unit 2401 is further configured to send first shuttling state information to the second terminal, where the first shuttling state information is used to indicate the start of shuttling.

[0403] Furthermore, the sending unit 2401 is also used to send data of the operation of moving the first cursor input by the user using the input device of the first terminal to the second terminal, so that the second terminal displays the animation of the movement of the first cursor on the display screen of the second terminal.

[0404] Furthermore, the dragging operation includes a pressing operation and a moving operation; the device may also include: an intercepting unit 2405.

[0405] The interception unit 2405 is used to intercept a pressing event after the user performs a pressing operation using the input device of the first terminal.

[0406] The sending unit 2401 is specifically configured to send the first operation parameter included in the pressing event to the second terminal.

[0407] The interception unit 2405 is further configured to intercept a movement event after the user uses the input device of the first terminal to perform a movement operation.

[0408] The sending unit 2401 is specifically configured to send the second operation parameter included in the movement event to the second terminal.

[0409] The first operation parameter and the second operation parameter are used for the second terminal to simulate a press event and a move event respectively, and then used to indicate initiation of dragging the object.

[0410] Furthermore, the interception unit 2405 is used to intercept a movement event after the user uses the input device of the first terminal to perform a movement operation.

[0411] The sending unit 2401 is specifically configured to send a third operation parameter included in the movement event to the second terminal, where the third operation parameter is used for the second terminal to simulate the movement event and further for instructing to move the first cursor.

[0412] Furthermore, the receiving unit 2402 is further configured to receive second shuttle status information from the second terminal, where the second shuttle status information is used to indicate the end of the shuttle.

[0413] Furthermore, the sending unit 2401 is further configured to send a request message to the second terminal, where the request message is used to request to drag data.

[0414] Furthermore, the device may also include: an input unit 2406.

[0415] The interception unit 2405 is used to cancel the interception of the input event.

[0416] The input unit 2406 is configured to receive a movement event after the user uses an input device of the first terminal to perform a movement operation.

[0417] The input unit 2406 is also used to generate a press event.

[0418] The display unit 2403 is specifically configured to display an animation of the object moving along with the second cursor on the display screen of the first terminal according to the moving event, the pressing event and the dragging data.

[0419] Furthermore, the display unit 2403 is also used to display an invisible window, the transparency of the invisible window is greater than a threshold, and the invisible window is used to initiate a drag event.

[0420] Further, the long press operation includes a press operation and a lift operation. The interception unit 2405 is used to intercept a press event after the user performs a press operation using the input device of the first terminal.

[0421] The sending unit 2401 is specifically configured to send the fourth operation parameter included in the pressing event to the second terminal.

[0422] The interception unit 2405 is further configured to intercept a lift event after the user performs a lift operation using the input device of the first terminal after performing a press operation for a period of time.

[0423] The sending unit 2401 is further configured to send the fifth operation parameter included in the lift event to the second terminal.

[0424] The fourth operation parameter and the fifth operation parameter are used for the second terminal to simulate a press event and a lift event respectively, and then used to trigger the windowing of the application interface.

[0425] The present application also provides a cross-device object dragging device, which can be applied to the first terminal or the second terminal in the above embodiment. The device may include: a processor, and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the various functions or steps performed by the PC, mobile phone or tablet computer in the above method embodiment when executing the instructions.

[0426] The embodiment of the present application also provides a terminal (the terminal may be the first terminal or the second terminal in the above embodiment), which may include: a display screen, a memory and one or more processors. The display screen, the memory and the processor are coupled. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the terminal can execute the various functions or steps performed by the PC or mobile phone or tablet computer in the above method embodiment. Of course, the terminal includes but is not limited to the above display screen, the memory and one or more processors. For example, the structure of the terminal can refer to Figure 2AThe structure of the mobile phone shown.

[0427] The present application also provides a chip system, which can be applied to the terminal (such as the first terminal or the second terminal) in the above embodiment. Fig.25 As shown, the chip system includes at least one processor 2501 and at least one interface circuit 2502. The processor 2501 may be a processor in the above-mentioned terminal. The processor 2501 and the interface circuit 2502 may be interconnected via a line. The processor 2501 may receive and execute computer instructions from the memory of the above-mentioned terminal via the interface circuit 2502. When the computer instructions are executed by the processor 2501, the terminal may execute the various steps executed by the PC, mobile phone or tablet computer in the above-mentioned embodiment. Of course, the chip system may also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0428] The embodiment of the present application also provides a computer-readable storage medium for storing computer instructions executed by the above-mentioned terminal (such as a PC or a mobile phone or a tablet computer).

[0429] The embodiment of the present application also provides a computer program product, including computer instructions executed by the above-mentioned terminal (such as a PC or a mobile phone or a tablet computer).

[0430] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0431] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0432] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0433] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0434] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0435] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A cross-device object dragging method, characterized in that: Applied to a second terminal, the second terminal is connected to the first terminal, the method includes: The second terminal creates a virtual input device; The second terminal receives first shuttle status information from the first terminal, where the first shuttle status information is used to indicate the start of the shuttle; In response to the first shuttle state information, the second terminal displays a first cursor on the second terminal; The second terminal receives an operation of moving the first cursor input by a user using an input device of the first terminal; In response to the operation of moving the first cursor, the second terminal displays an animation of the movement of the first cursor on a display screen of the second terminal until the first cursor moves onto the object; The second terminal displays the first cursor on the object displayed by the second terminal; the second terminal receives a first operation parameter from the first terminal, where the first operation parameter is an operation parameter included in a pressing event received by the first terminal after a user performs a pressing operation using an input device of the first terminal; The second terminal receives a second operation parameter from the first terminal, where the second operation parameter is an operation parameter included in a movement event received by the first terminal after a user uses an input device of the first terminal to perform a movement operation; The second terminal uses the virtual input device to simulate the pressing event according to the first operating parameter and simulates the moving event according to the second operating parameter, wherein the pressing event and the moving event are used to indicate initiation of dragging the object; In response to the pressing event and the moving event, the second terminal displays an animation of the object moving along with the first cursor on a display screen of the second terminal; When determining that the object is dragged out of the edge of the display screen of the second terminal, the second terminal sends the drag data to the first terminal.

2. The method according to claim 1, characterized in that The drag data is used by the first terminal to display an animation of the object moving along with the second cursor on a display screen of the first terminal.

3. The method according to claim 1, characterized in that The first operation parameter includes one or more of a mouse button flag, coordinate information, scroll wheel information, and key position information; The second operation parameter includes one or more of a mouse button flag, coordinate information, scroll wheel information, and key position information.

4. The method according to claim 1, characterized in that: The object is an icon of an application; or, the object is a window, and the window includes an interface of the application; When the object is the icon of the application, the drag data includes: the icon of the application; when the object is the window, the drag data includes: the interface of the application.

5. The method according to claim 4, characterized in that After sending the drag data to the first terminal, the method further includes: The second terminal receives an indication message from the first terminal, where the indication message is sent by the first terminal after a user inputs a drag-and-release operation using an input device of the first terminal, and is used to indicate that dragging of the object ends; The second terminal sends data to the first terminal, and the data is used by the first terminal to display the interface of the application on the display screen of the first terminal.

6. The method according to claim 4, characterized in that When the object is an icon of the application, after the second terminal receives the first operation parameter and the second operation parameter from the first terminal, the method further includes: The second terminal creates a drag object according to the icon of the application, and the drag object includes: an icon that is the same as the icon of the application; In response to the pressing event and the moving event, the second terminal displays an animation of the object moving along with the first cursor on a display screen of the second terminal, including: In response to the pressing event and the moving event, the second terminal displays an animation of the dragged object moving along with the first cursor.

7. The method according to claim 4, characterized in that When the object is the window, before the second terminal displays a first cursor on the object displayed on the second terminal, the method further includes: The second terminal displays the interface of the application; When the first cursor is displayed on the status bar of the second terminal, the second terminal receives a long press operation input by a user using an input device of the first terminal, so as to trigger the windowing of the interface of the application; In response to the long press operation, the second terminal displays the window; the window specifically includes a screenshot of the interface of the application, and the drag data specifically includes the screenshot of the interface of the application.

8. The method according to claim 1, characterized in that The second terminal displays, on a display screen of the second terminal, an animation of the object moving along with the first cursor, including: The second terminal displays an animation of the shadow of the object moving along with the first cursor on a display screen of the second terminal.

9. The method according to claim 8, characterized in that The object is a text, a file or a folder; the drag data includes the drag event content and the bitmap of the shadow; When the object is text, the drag event content includes the text; when the object is a file or folder, the drag event content is the file path.

10. The method according to claim 1, characterized in that After the second terminal displays an animation of the object moving along with the first cursor on a display screen of the second terminal in response to the pressing event and the moving event, the method further includes: The second terminal acquires and caches the drag data; The sending the drag data to the first terminal includes: The second terminal sends the buffered drag data to the first terminal.

11. The method according to claim 10, characterized in that Before sending the drag data to the first terminal, the method further includes: The second terminal receives a request message from the first terminal, where the request message is used to request the drag data.

12. The method according to claim 1, characterized in that The second terminal receives an operation of moving the first cursor input by a user using an input device of the first terminal, including: The second terminal receives a third operation parameter from the first terminal, where the third operation parameter is an operation parameter included in a movement event received by the first terminal after a user uses an input device of the first terminal to perform a movement operation; The second terminal simulates the movement event according to the third operation parameter, where the movement event is used to instruct to move the first cursor.

13. The method according to claim 1, characterized in that After the second terminal determines that the object is dragged out of the edge of the display screen of the second terminal, the method further includes: The second terminal sends second shuttling status information to the first terminal, where the second shuttling status information is used to indicate that the shuttling is finished.

14. The method according to claim 7, characterized in that The long press operation includes a press operation and a lift operation; The second terminal receives a long press operation input by a user using an input device of the first terminal, including: The second terminal receives a fourth operation parameter from the first terminal, where the fourth operation parameter is an operation parameter included in the pressing event received by the first terminal after the user performs the pressing operation using an input device of the first terminal when the first cursor is displayed on the status bar; The second terminal receives a fifth operation parameter from the first terminal, where the fifth operation parameter is an operation parameter included in a lift event received by the first terminal after the user performs the lift operation using an input device of the first terminal after performing the press operation for a period of time; The second terminal simulates the pressing event according to the fourth operation parameter, and simulates the lifting event according to the fifth operation parameter, wherein the pressing event and the lifting event are used to trigger the interface windowing of the application; In response to the long press operation, the second terminal displays the window, including: In response to the pressing event and the lifting event, the second terminal displays the window.

15. The method according to any one of claims 1 to 14, characterized in that The second terminal creates a virtual input device, including: After the second terminal successfully establishes a connection with the first terminal, creating a virtual input device; or, The second terminal receives a notification message from the first terminal, and in response to the notification message, the second terminal creates the virtual input device.

16. A method for dragging objects across devices, characterized in that: Applied to a first terminal, the first terminal is connected to a second terminal, the method comprises: the first terminal receives an operation of moving a second cursor input by a user using an input device of the first terminal; In response to the operation of moving the second cursor, the first terminal displays an animation of the movement of the second cursor on a display screen of the first terminal; Determining, by the first terminal, that the second cursor slides out of an edge of a display screen of the first terminal; The first terminal sends first shuttling state information to the second terminal, where the first shuttling state information is used to indicate the start of shuttling; The first terminal sends data of an operation of moving a first cursor input by a user using an input device of the first terminal to the second terminal, so that the second terminal displays an animation of the movement of the first cursor on a display screen of the second terminal; After the user performs a pressing operation using an input device of the first terminal, the first terminal intercepts the pressing event; The first terminal sends a first operation parameter included in the pressing event to the second terminal; After the user uses the input device of the first terminal to perform a movement operation, the first terminal intercepts the movement event; The first terminal sends a second operation parameter included in the movement event to the second terminal; The first operation parameter and the second operation parameter are respectively used by the second terminal to simulate the pressing event and the moving event using a virtual input device, and then used to instruct to initiate a drag on an object displayed by the second terminal, and the virtual input device is created by the second terminal; The first terminal receives drag data from the second terminal, wherein the drag data is sent to the first terminal after the second terminal determines that the object is dragged out of the edge of the display screen of the second terminal when the object moves with the first cursor on the display screen of the second terminal.

17. The method according to claim 16, characterized in that After the first terminal receives the drag data from the second terminal, the method further includes: The first terminal displays an animation of the object moving along with the second cursor on a display screen of the first terminal according to the drag data, the pressing event and the moving event.

18. The method according to claim 17, characterized in that The first operation parameter includes one or more of a mouse button flag, coordinate information, scroll wheel information, and key position information; The second operation parameter includes one or more of a mouse button flag, coordinate information, scroll wheel information, and key position information.

19. The method according to claim 16, characterized in that The object is an icon of an application; or, the object is a window, and the window includes an interface of the application; When the object is the icon of the application, the drag data includes: the icon of the application; when the object is the window, the drag data includes: the interface of the application.

20. The method according to claim 19, characterized in that After the first terminal receives the drag data from the second terminal, the method further includes: The first terminal receives a drag-and-release operation input by a user using an input device of the first terminal; In response to the drag-and-release operation, the first terminal displays an interface of the application on a display screen of the first terminal.

21. The method according to claim 20, characterized in that The first terminal displays the interface of the application on a display screen of the first terminal, including: The first terminal sends an indication message to the second terminal, where the indication message is used to indicate that the dragging of the object ends; The first terminal receives data from the second terminal; The first terminal displays the interface of the application on a display screen of the first terminal according to the data.

22. The method according to claim 19, characterized in that When the object is the window, before the first terminal sends the first operation parameter included in the press event and the second operation parameter included in the move event to the second terminal, the method further includes: The first terminal sends data of a long press operation input by a user using an input device of the first terminal to the second terminal, where the long press operation is input by the user when the first cursor is displayed on a status bar of the second terminal, and is used to trigger the windowing of the interface of the application displayed by the second terminal; the window specifically includes a screenshot of the interface of the application, and the drag data specifically includes the screenshot of the interface of the application.

23. The method according to claim 17, characterized in that The first terminal displays an animation of the object moving along with the second cursor on a display screen of the first terminal according to the drag data, the press event and the move event, including: The first terminal displays an animation of the shadow of the object moving along with the second cursor on a display screen of the first terminal according to the drag data, the press event and the move event.

24. The method according to claim 23, characterized in that The object is a text, a file or a folder; the drag data includes the drag event content and the bitmap of the shadow; When the object is text, the drag event content includes the text; when the object is a file or folder, the drag event content is the file path.

25. The method according to claim 16, characterized in that The first terminal sends data of an operation of moving the first cursor input by a user using an input device of the first terminal to the second terminal, including: After the user uses the input device of the first terminal to perform a movement operation, the first terminal intercepts the movement event; The first terminal sends a third operation parameter included in the movement event to the second terminal, where the third operation parameter is used by the second terminal to simulate the movement event and further to instruct to move the first cursor.

26. The method according to claim 17, characterized in that Before the first terminal displays an animation of the object moving with the second cursor on a display screen of the first terminal according to the drag data, the press event and the move event, the method further includes: The first terminal receives second shuttling status information from the second terminal, where the second shuttling status information is used to indicate that shuttling is completed.

27. The method according to claim 16, characterized in that Before the first terminal receives the drag data from the second terminal, the method further includes: The first terminal sends a request message to the second terminal, where the request message is used to request the drag data.

28. The method according to claim 26, characterized in that After the first terminal receives the second shuttling state information from the second terminal, the method further includes: The first terminal cancels interception of the input event; The first terminal displays an animation of the object moving along with the second cursor on a display screen of the first terminal according to the drag data, the press event, and the move event, including: After the user performs a movement operation using an input device of the first terminal, the first terminal receives a movement event; The first terminal generates a press event; The first terminal displays an animation of the object moving along with the second cursor on a display screen of the first terminal according to the moving event, the pressing event and the dragging data.

29. The method according to claim 16, characterized in that The method further comprises: The first terminal displays an invisible window, the transparency of the invisible window is greater than a threshold, and the invisible window is used to initiate a drag event.

30. The method according to claim 22, characterized in that The long press operation includes a press operation and a lift operation; The first terminal sends data of a long press operation input by a user using an input device of the first terminal to the second terminal, including: After the user performs the pressing operation using the input device of the first terminal, the first terminal intercepts the pressing event, and the first terminal sends a fourth operation parameter included in the pressing event to the second terminal; After the user performs the lifting operation using the input device of the first terminal after performing the pressing operation for a period of time, the first terminal intercepts the lifting event, and the first terminal sends a fifth operation parameter included in the lifting event to the second terminal; The fourth operation parameter and the fifth operation parameter are used by the second terminal to simulate the pressing event and the lifting event respectively, so as to trigger the windowing of the interface of the application.

31. A cross-device object dragging device, characterized in that: include: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to implement the method according to any one of claims 1-15 or the method according to any one of claims 16-30 when executing the instructions.

32. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 15 is implemented, or the method according to any one of claims 16 to 30 is implemented.

33. A cross-device object dragging system, characterized in that: include: A first terminal and a second terminal; the first terminal is connected to the second terminal; The second terminal is used to create a virtual input device; receiving first shuttle state information from the first terminal, the first shuttle state information being used to indicate the start of shuttling; and displaying a first cursor on the second terminal in response to the first shuttle state information; receiving an operation of moving the first cursor input by a user using an input device of the first terminal; In response to the operation of moving the first cursor, displaying an animation of the movement of the first cursor on the display screen of the second terminal until the first cursor moves onto the object; and displaying the first cursor on the object displayed on the second terminal; receiving a first operation parameter from the first terminal, the first operation parameter being an operation parameter included in a press event received by the first terminal after a user performs a press operation using an input device of the first terminal; receiving a second operation parameter from the first terminal, the second operation parameter being an operation parameter included in a move event received by the first terminal after a user performs a move operation using the input device of the first terminal; Using the virtual input device to simulate the pressing event according to the first operating parameter and to simulate the moving event according to the second operating parameter, the pressing event and the moving event being used to indicate initiation of dragging the object; In response to the pressing event and the moving event, displaying an animation of the object moving along with the first cursor on the display screen of the second terminal; and sending drag data to the first terminal when it is determined that the object is dragged out of the edge of the display screen of the second terminal; The first terminal is configured to receive an operation of moving a second cursor input by a user using an input device of the first terminal; in response to the operation of moving the second cursor, display an animation of the movement of the second cursor on a display screen of the first terminal; determine that the second cursor slides out of an edge of the display screen of the first terminal; and send first shuttle state information to the second terminal, wherein the first shuttle state information is used to indicate the start of shuttle; sending data of an operation of moving the first cursor input by a user using an input device of the first terminal to the second terminal, so that the second terminal displays an animation of the movement of the first cursor on a display screen of the second terminal; and intercepting a pressing event after the user performs the pressing operation using the input device of the first terminal; Sending a first operation parameter included in the pressing event to the second terminal; After the user uses the input device of the first terminal to perform the movement operation, intercepting the movement event; sending a second operation parameter included in the moving event to the second terminal; wherein the first operation parameter and the second operation parameter are respectively used by the second terminal to simulate the pressing event and the moving event by using a virtual input device, and then used to indicate initiating dragging of an object displayed on the second terminal, wherein the virtual input device is created by the second terminal; receiving drag data from the second terminal, wherein the drag data is sent to the first terminal by the second terminal after determining that the object is dragged out of the edge of the display screen of the second terminal during the process in which the object moves on the display screen of the second terminal along with the first cursor.

34. The system according to claim 33, characterized in that The drag data is used by the first terminal to display an animation of the object moving along with the second cursor on a display screen of the first terminal.

35. The system according to claim 33, characterized in that The first operation parameter includes one or more of a mouse button flag, coordinate information, scroll wheel information, and key position information; The second operation parameter includes one or more of a mouse button flag, coordinate information, scroll wheel information, and key position information.

36. The system according to claim 33, characterized in that The step of displaying the animation of the object moving along with the first cursor on the display screen of the second terminal includes: An animation of the shadow of the object moving along with the first cursor is displayed on the display screen of the second terminal.

37. The system according to claim 36, characterized in that The object is a text, a file or a folder; the drag data includes the drag event content and the bitmap of the shadow; When the object is text, the drag event content includes the text; when the object is a file or folder, the drag event content is the file path.

38. The system according to claim 33, characterized in that In response to the pressing event and the moving event, after displaying the animation of the object moving along with the first cursor on the display screen of the second terminal, the second terminal is further used to obtain and cache the drag data; The sending the drag data to the first terminal includes: sending the buffered drag data to the first terminal.

39. The system according to claim 33, characterized in that The receiving an operation of moving the first cursor input by a user using an input device of the first terminal includes: receiving a third operation parameter from the first terminal, where the third operation parameter is an operation parameter included in a movement event received by the first terminal after a user uses an input device of the first terminal to perform a movement operation; The moving event is simulated according to the third operating parameter, where the moving event is used to instruct moving the first cursor.

40. The system of claim 33, wherein: The second terminal is further configured to send second shuttle status information to the first terminal after determining that the object is dragged out of the edge of the display screen of the second terminal, wherein the second shuttle status information is used to indicate the end of the shuttle.

41. The system according to any one of claims 33 to 40, characterized in that The step of creating a virtual input device comprises: After the connection with the first terminal is successfully established, creating a virtual input device; or, A notification message is received from the first terminal, and in response to the notification message, the virtual input device is created.

Images