Keyboard and mouse crossing method and terminal equipment

Abstract

The invention provides a keyboard and mouse crossing method and terminal equipment. The method provided by the invention comprises the following steps: a first device displays a cursor corresponding to an input device on a first screen; the first device detects a first operation acting on the input device, wherein the first operation is used for dragging a cursor to determine a first direction as a connection direction; in response to the first operation, the first device establishes a connection with a second device in a first direction of the first device; the first device detects a second operation acting on the input device, wherein the second operation is used for dragging a cursor displayed on the first screen in the first direction; in response to the second operation, the first equipment controls the cursor to move out of the first screen, and enters a second screen of the second equipment to be displayed; wherein in response to a third operation acting on the input device, the second device executes the control instruction corresponding to the third operation, and the third operation is used for inputting the control instruction on the second device.

Description

Keyboard and mouse traversal method and terminal device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 2, 2022, with application number 202211071888.9 and application name “Keyboard and mouse crossing method and terminal device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a keyboard and mouse traversal method and a terminal device. Background Art

[0003] With the development of terminal technology, the types and number of terminal devices are increasing. These devices include mobile phones, tablets, computers, and wearable devices. Currently, a single terminal device is no longer sufficient for users. Users can use multiple devices, such as mobile phones, tablets, and computers, to meet their work and study needs. However, when multiple devices have services that require user processing, users need to switch back and forth between different terminal devices, which is cumbersome and detrimental to the user experience.

[0004] Summary of the Invention

[0005] The present application provides a keyboard and mouse traversal method and terminal device, which can operate different terminal devices through the mouse and keyboard to achieve seamless access between different terminal devices.

[0006] In a first aspect, a keyboard and mouse crossing method is provided, which is applied to a first device, which can also be called a terminal device, and the first device is connected to an input device for inputting control instructions, and the first device includes a first screen; the method also includes: the first device displays a cursor corresponding to the input device on the first screen; the first device detects a first operation acting on the input device, and the first operation is used to drag the cursor to determine a first direction as a connection direction; in response to the first operation, the first device establishes a connection with a second device located in the first direction of the first device; the first device detects a second operation acting on the input device, and the second operation is used to drag the cursor displayed on the first screen in the first direction; in response to the second operation, the first device controls the cursor to move out of the first screen and enter the second screen of the second device for display; wherein, in response to a third operation acting on the input device, the second device executes a control instruction corresponding to the third operation, and the third operation is used to input a control instruction on the second device.

[0007] The input device may be a keyboard and a mouse, which are referred to as keyboard and mouse in the present embodiment. The cursor may also be referred to as a mouse pointer, which is not limited in the present embodiment.

[0008] The first device and the second device can establish a connection via WiFi or Bluetooth, where WiFi can include a local area network or peer-to-peer (P2P). The first device and the second device establish a connection, which can be understood as the first device being able to communicate with the second device. The first device can be the computer 101 in the embodiment, and the second device can be the mobile phone 102 or tablet 103 in the embodiment, but the present application is not limited thereto.

[0009] After the first device and the second device establish a connection, the first device and the second device can support keyboard and mouse crossing, that is, the cursor corresponding to the input device can move from the first screen of the first device to the second screen of the second device, and can also move from the second screen of the second device to the first screen of the first device.

[0010] The first operation may be that the cursor touches the side edge of the first screen twice in succession. The user controls the cursor to touch the side edge of the first screen twice in succession through the input device, triggering the first device to connect with other devices. If the side edge of the first screen is right, the first device is connected to the device on the right, and if the side edge of the first screen is left, the first device is connected to the device on the left, so the first operation can also be used to trigger the first device to connect with other devices on the touched edge side. The direction of the side edge triggered by the first operation can be called the first direction. The first device detects the first operation, and in response to the first operation, can connect with the device in the first direction. If the second device is in the first direction of the first device, the first device can connect with the second device.

[0011] The second operation is used to drag the cursor displayed on the first screen in the first direction. That is, the user controls the input device to drag the cursor toward the second device to move it to the second screen of the second device. The second operation may also be referred to as a traversal operation, which is not limited in this application. In response to the second operation, the first device controls the cursor to move out of the first screen and enter the second screen of the second device for display.

[0012] After the cursor moves to the second screen, the user can control the input device to perform various operations on the second device, such as clicking, dragging, or input operations, etc. The second device executes a control instruction corresponding to the third operation.

[0013] For example, in the example shown in Figure 2, the first device is a computer 101, the second device is a mobile phone 102, the input device is a keyboard and mouse, the mouse pointer is the cursor of the input device, and the cursor of the input device moves from the first device to the second device. The user can use the cursor of the input device to open the memo and edit the memo on the second device.

[0014] The keyboard and mouse crossing method provided in this application allows input devices to cross between different devices. A set of input devices can be used to control different devices. There is no need to switch the operation mode back and forth between different terminal devices, which avoids cumbersome operations and is conducive to improving user experience.

[0015] In combination with the first aspect, in certain implementations of the first aspect, in response to a first operation, a first device establishes a connection with a second device located in a first direction of the first device, including: in response to the first operation, the first device finds a second device in the first direction that meets the connection conditions; the first device establishes a connection with the second device.

[0016] In response to the first operation, the first device determines whether there is a device that meets the connection condition in the first direction. If there is a second device that meets the connection condition in the first direction, the first device may establish a connection with the second device.

[0017] The keyboard and mouse crossing method provided in this application, the first operation triggers the first device to automatically search for a device that meets the connection conditions and connects to the device that meets the connection conditions, providing a method for automatically connecting devices, which is conducive to reducing user operations and improving user operation efficiency.

[0018] In combination with the first aspect, in certain implementations of the first aspect, before the first device detects the first operation acting on the input device, the method also includes: the first device performs device discovery; the first device stores the device identification of the device that meets the connection condition and the direction of the device that meets the connection condition relative to the first device in a device table; in response to the first operation, the first device finds the second device that meets the connection condition in the first direction, including: in response to the first operation, the first device finds the second device that meets the connection condition in the first direction through the device table.

[0019] After the first device is powered on, it can perform device discovery and store the device identifiers that meet the connection conditions and the directions of the devices that meet the connection conditions relative to the first device in a device table. The device table may also include information such as the direction of travel between the first device and the device that meets the connection conditions, and the angle of the device that meets the connection conditions relative to the first device, although this is not limited in the present embodiment. The device table may be the tB table in the embodiment.

[0020] In response to the first operation, the first device may search the device table for a second device that meets the connection condition in the first direction.

[0021] The keyboard and mouse traversal method provided in this application records the device information of devices that meet the connection conditions through a device table. In subsequent use, there is no need to perform device discovery, and it can quickly determine whether there is a second device that meets the connection conditions in the first direction, which can improve efficiency.

[0022] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the first device detects a change in the direction of the second device relative to the first device; and the first device updates the direction of the second device relative to the first device in the device table.

[0023] The keyboard and mouse traversal method provided in the present application can update the information in the device table in real time when a change in the direction of the second device is detected, which is conducive to improving the accuracy of the keyboard and mouse traversal.

[0024] In combination with the first aspect, in some implementations of the first aspect, a third device is also included in the first direction of the first device. Before the first device establishes a connection with the second device, the method also includes: the first device detects a fourth operation of the user, and the fourth operation is used to select the second device.

[0025] If there are multiple devices that meet the connection condition in the first direction of the first device, for example, the second device and the third device, the first device can determine the device to connect to based on the user's selection operation. If the user selects the second device, the first device can connect to the second device in response to the operation.

[0026] The keyboard and mouse crossing method provided in this application allows users to independently select the connected device, which is highly flexible and conducive to improving user experience.

[0027] In combination with the first aspect, in certain implementations of the first aspect, before the first device detects a first operation acting on the input device, the method also includes: the first device displays a first interface, the first interface includes an automatic connection option; in response to the operation acting on the automatic connection option, the automatic connection option is selected.

[0028] The above method may be a method executed by the first device when the automatic connection option is turned on. The automatic connection option may be selected by a user operation or by default, which is not limited in the embodiments of the present application.

[0029] The automatic connection option can be as shown in Figure 5 of the embodiment. The automatic connection option is to touch the mouse pointer on the side edge of the computer screen twice in succession to establish a connection, but the present application is not limited to this.

[0030] It is understood that when the automatic connection option is selected, the first device automatically connects to the device that meets the connection conditions. When the automatic connection option is not selected, the first device manually connects to the device that meets the connection conditions.

[0031] The keyboard and mouse crossing method provided in this application provides users with a visual interface, and the user determines the connection method, which can increase the flexibility of the connection and provide users with multiple implementation methods to meet the needs of users in different situations, which is conducive to improving user experience.

[0032] In combination with the first aspect, in some implementations of the first aspect, when the automatic connection option is not selected, the method also includes: in response to an operation of opening a first application on a first device, the first device displays a second interface, the second interface including: a first icon for representing the first device, a second icon for representing the second device, and placement areas located on both sides of the first icon; in response to a fifth operation of dragging the second icon to the first placement area, the first device establishes a connection with the second device, and the first placement area is in a first direction of the first device.

[0033] When the automatic connection option is not selected, the first device provides a visual interface, ie, a second interface, to the user, and the user can select a device to connect to in the visual interface.

[0034] The first application is an application that provides a visual interface. For example, the first application may be a computer manager. The second interface may be a super keyboard and mouse interface. The second interface and the first interface may be the same interface or may be different interfaces, and this embodiment of the application is not limited to this.

[0035] The first device detects the fifth operation of the user dragging the second icon to the first placement area, which may indicate that the user has selected the second device to connect with the first device. The first device may establish a connection with the second device in response to the fifth operation of dragging the second icon to the first placement area.

[0036] The keyboard and mouse crossing method provided in this application provides users with a way to manually connect devices, which can meet the user's needs for independent device selection and is conducive to improving user experience.

[0037] In combination with the first aspect, in certain implementations of the first aspect, in response to the fifth operation of dragging the second icon to the first placement area, the first device establishes a connection with the second device, including: in response to the fifth operation, the first device determines whether there is a connected device in the first direction; if there is no connected device, the first device establishes a connection with the second device; if there is a connected device, prohibiting the first device from connecting to the second device, and restoring the position of the second icon to the position before dragging.

[0038] If there is a device in the first placement area, it may mean that there is a connected device in the first direction of the first device. If there is no device in the first placement area, it may mean that there is no connected device in the first direction of the first device.

[0039] If there is a connected device, the first device may prohibit the first device from connecting to the second device and not accept the fifth operation, ie, restore the position of the second icon to the position before dragging.

[0040] The keyboard and mouse traversal method provided in the present application can determine whether there is a connected device before connecting in a manual device connection mode, thereby avoiding the scenario of connecting two devices in one direction.

[0041] In combination with the first aspect, in some implementations of the first aspect, the connection condition is that the login account of the second device is the same as that of the first device.

[0042] The second device uses the same login account as the first device, and the first and second devices can discover each other, forming a trust loop. After the first and second devices establish a connection, they can communicate.

[0043] The keyboard and mouse crossing method provided in this application allows different devices logged into the same account to be connected, which is conducive to multiple devices serving one user at the same time, helping to meet user needs and improve user experience.

[0044] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: in response to the second operation, the first device obtains the exit position of the cursor moving out of the first screen; the first device determines the initial position of the cursor entering the second screen based on the exit position; the first device sends the initial position to the second device to instruct the second device to display the cursor at the initial position of the second screen.

[0045] In response to the second operation, the first device controls the cursor to move out of the first screen and enter the second screen of the second device for display, and obtains the exit position of the cursor when it moves out of the first screen. The exit position may also be referred to as a crossing position, which is not limited in the embodiments of the present application. The exit position, that is, the crossing position of the mouse on the screen, can be the coordinates of the mouse on the screen.

[0046] The first device may calculate an initial position of the cursor on the second screen based on the exit position, that is, an initial position of the cursor after crossing.

[0047] The keyboard and mouse crossing method provided in this application can calculate the initial position of the cursor on the second screen based on the exit position, which is conducive to making the initial position consistent with the exit position, and automatically adjusting the display position of the cursor according to different devices, which is conducive to improving the user experience.

[0048] In combination with the first aspect, in certain implementations of the first aspect, the first device determines the initial position of the cursor entering the second screen based on the exit position, including: the first device determines the initial position based on the exit position, the screen resolution of the first screen, and the screen resolution of the second screen.

[0049] It is understandable that the screen resolution of the first screen is different from the screen resolution of the second screen.

[0050] The first device may determine an initial position adapted to the moved-out position based on a difference in screen resolution.

[0051] The keyboard and mouse crossing method provided in this application determines the initial position adapted to the moved-out position based on the difference in screen resolution, which is helpful in offsetting the different display positions of the cursor caused by different screen resolutions, and realizes automatic adjustment of the display position of the cursor according to different devices, which is helpful in improving the user experience.

[0052] In combination with the first aspect, in certain implementations of the first aspect, the first device determines the initial position based on the exit position, the screen resolution of the first screen, and the screen resolution of the second screen, including: the first device determines the first conversion rate according to the screen resolution of the first screen and the screen resolution of the second screen; the first device determines the initial position based on the exit position and the first conversion rate.

[0053] The first conversion rate can be the ratio of the screen resolution of the first screen to the screen resolution of the second screen, and the difference in screen resolution between different devices is reflected by the ratio. The first conversion rate can be the ratio of the screen resolution of the second screen to the screen resolution of the first screen, which is not limited in the embodiments of the present application. The first conversion rate can also be called the sensitivity conversion rate.

[0054] The first device may determine a product of the moved-out position and the first conversion rate as the initial position.

[0055] The keyboard and mouse crossing method provided in this application reflects the difference in resolution of different screens through the first conversion rate, and then determines the initial position through the exit position and the first conversion rate. It can balance the difference between the initial position and the exit position, which is conducive to making the initial position and the exit position consistent.

[0056] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: the first device obtains a first movement speed of the cursor on the first screen in response to a second operation; the first device determines a second movement speed of the cursor on the second screen based on the first movement speed; and the first device sends the second movement speed to the second device to instruct the second device to move the cursor at the second movement speed on the second screen.

[0057] Different screen resolutions will also result in different mouse movement speeds. Therefore, this application can also determine the second movement speed of the cursor on the second screen based on the first movement speed of the cursor on the first screen to ensure that the mouse movement speed is consistent when traveling between different devices.

[0058] The first movement speed and the second movement speed may also be referred to as mouse sensitivity, which is not limited in the embodiments of the present application. The second movement speed may also be referred to as the movement speed of the mouse after crossing. The first movement speed and the second movement speed are displacements per unit time, which can also be understood as the number of pixels moved per unit time. The unit time may be preset, for example, 3 milliseconds, 4 milliseconds, or 20 milliseconds, which is not limited in the embodiments of the present application.

[0059] The keyboard and mouse crossing method provided in this application can calculate the second movement speed of the cursor on the second screen based on the first movement speed of the cursor on the first screen, which is conducive to making the first movement speed and the second movement speed consistent, and realizing automatic adjustment of the cursor movement speed according to different devices, which is conducive to improving user experience.

[0060] In combination with the first aspect, in certain implementations of the first aspect, the first device determines the second movement speed of the cursor on the second screen based on the first movement speed, including: the first device determines the second movement speed based on the first movement speed, the screen resolution of the first screen, and the screen resolution of the second screen.

[0061] It is understandable that the screen resolution of the first screen is different from the screen resolution of the second screen.

[0062] The first device may determine a second movement speed adapted to the first movement speed based on the difference in screen resolution.

[0063] The keyboard and mouse crossing method provided in this application determines a second movement speed that is adapted to the first movement speed based on the difference in screen resolution, which is beneficial to offset the different cursor movement speeds caused by different screen resolutions, and automatically adjusts the cursor movement speed according to different devices, which is beneficial to improving user experience.

[0064] In combination with the first aspect, in certain implementations of the first aspect, the first device determines the second moving speed based on the first moving speed, the screen resolution of the first screen, and the screen resolution of the second screen, including: the first device determines the second conversion rate according to the screen resolution of the first screen and the screen resolution of the second screen; the first device determines the second moving speed based on the first moving speed and the second conversion rate.

[0065] The second conversion rate can be the ratio of the screen resolution of the first screen to the screen resolution of the second screen, and the difference in screen resolution between different devices is reflected by the ratio. The second conversion rate can also be the ratio of the screen resolution of the second screen to the screen resolution of the first screen, which is not limited in the embodiments of the present application. The second conversion rate can also be called the sensitivity conversion rate.

[0066] The first device may determine a product of the first movement speed and the second conversion rate as the second movement speed.

[0067] The keyboard and mouse crossing method provided in this application reflects the difference in resolution of different screens through the second conversion rate, and then determines the second movement speed through the first movement speed and the second conversion rate. This can balance the difference between the first movement speed and the second movement speed, which is conducive to making the first movement speed and the second movement speed consistent.

[0068] In combination with the first aspect, in certain implementations of the first aspect, after the first device controls the cursor to move out of the first screen and enter the second screen of the second device for display, the method also includes: the first device receives a third operation; the first device intercepts an input event corresponding to the third operation; the first device sends the input event to the second device, wherein the second device determines the control instruction corresponding to the third operation based on the input event.

[0069] In response to the third operation, the first device controls the cursor to move out of the first screen and onto the second screen of the second device for display. At this time, the input device can input a control instruction on the second device. When the user inputs a control instruction on the second device through the input device, the first device can receive an input event corresponding to the user's operation, intercept the input event, and send an input event corresponding to the third operation to the second device to instruct the second device to execute the control instruction corresponding to the third operation. The input event can be a click, double-click, drag, or move. The control instruction refers to a process triggered by a click, double-click, drag, or move.

[0070] The keyboard and mouse crossing method provided in the present application can intercept the input event corresponding to the operation acting on the input device when the cursor corresponding to the input device connected to the first device moves to the second device, and send the input event to the second device. The second device executes the control instruction corresponding to the operation acting on the input device. This can realize the sharing of input devices by different devices, and there is no need to switch different operation modes between different devices, thus avoiding cumbersome operations and improving user experience.

[0071] In combination with the first aspect, in certain implementations of the first aspect, after the first device controls the cursor to move out of the first screen and display on the second screen of the second device in response to the second operation, the method also includes: the first device detects a sixth operation acting on the input device, and the sixth operation is used to drag the cursor displayed on the second screen toward the first device; in response to the sixth operation, the first device controls the cursor to move out of the second screen and display on the first screen of the second device; the first device stops intercepting the input event corresponding to the operation acting on the input device to execute the control instruction corresponding to the operation of the input device.

[0072] The sixth operation is used to drag the cursor displayed on the second screen toward the first device. That is, the user controls the input device to move the cursor from the second screen to the first screen. In response to the sixth operation, the first device controls the cursor to move out of the second screen and onto the first screen. At this time, the input device can input control instructions on the first device. When the user inputs control instructions on the first device through the input device, the first device can stop intercepting input events corresponding to the user's operation and, upon receiving input events corresponding to the user's operation, can execute the control instructions corresponding to the user's operation.

[0073] The keyboard and mouse crossing method provided in the present application is that when the cursor corresponding to the input device connected to the first device moves back (or passes back) from the second device to the first device, the first device can stop intercepting the input event corresponding to the operation acting on the input device and execute the control instruction corresponding to the operation acting on the input device, so that the input device can switch back and forth between different devices and control the device switched to.

[0074] In combination with the first aspect, in certain implementations of the first aspect, after the first device controls the cursor to move out of the first screen and enter the second screen of the second device for display in response to the second operation, the method also includes: the first device receives a seventh operation acting on the input device, and the seventh operation is used to input a control instruction for forcibly reclaiming the cursor on the second device; in response to the seventh operation, the first device controls the cursor to be displayed on the first screen; the first device stops intercepting the input event corresponding to the operation acting on the input device to execute the control instruction corresponding to the operation of the input device.

[0075] When the cursor of the input device is displayed on the second device, the user can use a shortcut key to forcibly recapture the input device, that is, to display the cursor of the input device on the first device. When the first device receives the user's operation of triggering the shortcut key to forcibly recapture the input device (that is, the seventh operation described above), in response to the user's operation, the cursor corresponding to the input device can be displayed in a specified position on the screen of the first device. The shortcut key for forcibly recapture the input device can be Esc or Ctrl+Alt+DELETE, and the specified position on the screen can be the center of the screen, but the present application is not limited to this.

[0076] When the cursor of the input device is displayed on the first device, the input device can input a control instruction on the first device. When the user inputs the control instruction on the first device through the input device, the first device can stop intercepting input events corresponding to the user's operation and, upon receiving the input event corresponding to the user's operation, can execute the control instruction corresponding to the user's operation.

[0077] The keyboard and mouse crossing method provided in this application can support users to force the cursor moved to other devices to move back to the first device through a control command without the need for the user to move, which can improve efficiency. In addition, it can be applied to more scenarios, such as scenarios where the user cannot see the cursor, etc., which can meet the different needs of users and improve user experience.

[0078] In combination with the first aspect, in certain implementations of the first aspect, the first direction is left or right, and after the first device establishes a connection with the second device located in the first direction of the first device, the method also includes: the first device detects that the second device moves from the first direction to a second direction opposite to the first direction; in response to an eighth operation, the first device controls the cursor to move out of the first screen and enter the second screen of the second device for display, and the eighth operation is opposite to the direction of dragging the cursor in the second operation.

[0079] The second device moves from a first direction to a second direction opposite to the first direction. That is, the second device moves from the first direction of the first device to the second direction of the second device. If the first direction is left, the second direction is opposite to the first direction and is right. If the first direction is right, the second direction is opposite to the first direction and is left.

[0080] The second device is in the second direction of the first device. If the cursor is moved out of the first screen and displayed on the second screen, the eighth operation can be used to drag the cursor displayed on the first screen in the second direction, which is opposite to the direction of dragging the cursor in the second operation.

[0081] The keyboard and mouse crossing method provided in this application can automatically detect the direction of the second device relative to the first device. If the direction of the second device relative to the first device is left or right, and the direction of the second device relative to the first device is changed to the opposite direction, the direction of the cursor moving from the first device to the second device is also changed to the opposite direction, which conforms to the user's usage habits and can improve the user experience.

[0082] In combination with the first aspect, in certain implementations of the first aspect, the first direction is left or right, and after the first device establishes a connection with the second device located in the first direction of the first device, the method also includes: the first device detects that the second device moves from the first direction to the middle; in response to a ninth operation, the first device controls the cursor to move out of the first screen and enter the second screen of the second device for display, and the ninth operation is in the same direction as the second operation of dragging the cursor.

[0083] The second device moving from the first direction to the middle may include: the second device moving from the left to the middle, and the second device moving from the right to the middle.

[0084] When the second device is in the middle of the first device and the direction of the first device is changed to the middle, if the cursor is moved out of the first screen and displayed on the second screen, the ninth operation can be used to drag the cursor displayed on the first screen in the first direction, which is the same as the direction of dragging the cursor in the second operation. That is to say, when the second device is in the middle of the first device and the direction of the cursor crossing from the first device to the second device remains unchanged.

[0085] The keyboard and mouse crossing method provided in this application can automatically detect the direction of the second device relative to the first device. If the direction of the second device relative to the first device is changed to the middle, the direction of the cursor moving from the first device to the second device remains unchanged.

[0086] In combination with the first aspect, in certain implementations of the first aspect, the first direction is center, and after the first device establishes a connection with the second device located in the first direction of the first device, the method also includes: the first device detects that the second device moves from the first direction to the second direction, and the second direction is left or right; in response to the tenth operation, the first device controls the cursor to move out of the first screen and enter the second screen of the second device for display, and the tenth operation is the same as the second direction.

[0087] The second device moves from the first direction to the second direction, which may include: the second device moves from the center to the left, and the second device moves from the center to the right.

[0088] The second device moves from the center to the left or right. If the cursor is moved out of the first screen and displayed on the second screen, the tenth operation can be used to drag the cursor displayed on the first screen to the left or right (i.e., the second direction), which is the same as the tenth operation and the second direction.

[0089] The keyboard and mouse crossing method provided in this application can automatically detect the direction of the second device relative to the first device. If the direction of the second device relative to the first device changes from left to right, the direction in which the cursor moves from the first device to the second device is the same as the second direction.

[0090] In combination with the first aspect, in certain implementations of the first aspect, the first direction is left or right, and the first device is connected to a third device located in the second direction of the first device, and the second direction is opposite to the first direction; after the first device establishes a connection with the second device located in the first direction of the first device, the method further includes: the first device detects that the second device moves from the first direction to the second direction; in response to an eleventh operation, the first device controls the cursor to move out of the first screen and enter the second screen of the second device for display, and the eleventh operation is the same as the direction of dragging the cursor in the second operation.

[0091] The first device is connected to the second device in the first direction of the first device, and is connected to the third device in the second direction of the first device. If the direction of the second device relative to the first device is changed to the second direction, since there is the third device in the second direction of the first device, the direction of the cursor from the first device to the second device remains unchanged, that is, the direction of the cursor dragged in the eleventh operation is the same as that in the second operation.

[0092] For example, in the example shown in FIG1 of the embodiment, the first device is a computer 101, the second device is a mobile phone 102, and the third device is a tablet 103. When the computer 101, mobile phone 102, and tablet 103 implement keyboard and mouse crossover, the mobile phone 102 and tablet 103 are located on the right and left sides of the computer 101. If the position of the mobile phone 102 changes and changes to the same side as the tablet 103, or the position of the tablet 103 changes and changes to the same side as the mobile phone 102, the computer 101 does not change the original mouse crossover direction.

[0093] In combination with the first aspect, in certain implementations of the first aspect, before the first device detects the first operation acting on the input device, the method also includes: the first device discovers the second device; the first device obtains the device identification of the second device and the device angle of the second device relative to the first device; the first device determines the coordinate direction and crossing direction of the second device based on the device angle; the first device records the device identification, device angle, coordinate direction and crossing direction in the device table.

[0094] The first device may perform device discovery and discover the presence of a second device. The first device may obtain a device identifier for the second device. The first device may use ultrasonic technology to determine the device angle of the second device relative to the first device. The device angle may also be referred to as a coordinate angle, which is not limited in this embodiment of the present application.

[0095] The first device can determine the direction of the second device relative to the first device, i.e., the coordinate direction, based on the device angle of the second device relative to the first device. The first device can determine the traversal direction, i.e., the direction in which the cursor moves from the first device to the second device, based on the coordinate direction.

[0096] For example, the device angle of the second device relative to the first device is 60°, the coordinate direction of the second device is left, and the crossing direction is left.

[0097] The first device may save the device information of the second device, including the device identification, device angle, coordinate direction, and traversing direction, into the device table.

[0098] The keyboard and mouse traversal method provided in this application can record the information of the discovered devices through the device table, which is conducive to subsequent rapid information judgment or connection.

[0099] In combination with the first aspect, in certain implementations of the first aspect, after the first device establishes a connection with the second device located in the first direction of the first device, the method also includes: the first device detects that the second device is moving; when the first device detects that the second device is stationary, it continues to detect the device angle of the second device after the movement; if the device angle after the movement and the device angle before the movement do not belong to the same angle range, the first device determines the coordinate direction after the movement and the crossing direction after the movement based on the device angle after the movement, wherein each coordinate direction corresponds to a preset angle range; the first device updates the device table based on the coordinate direction after the movement and the crossing direction after the movement.

[0100] The second device can sense movement through a sensor such as an accelerometer and send the movement information to the first device. The first device can detect the movement of the second device. When the first device detects that the second device is stationary, the first device can continue to obtain the device angle of the second device after the movement, and determine the coordinate direction after the movement and the crossing direction after the movement based on the device angle after the movement.

[0101] The first device can determine whether the coordinate direction after the movement has changed by determining whether the device angle after the movement and the device angle before the movement belong to the same angle range, and further determine whether the crossing direction has changed based on the coordinate direction.

[0102] If the device angle after the movement is different from the device angle before the movement, it can be indicated that the coordinate direction has changed, and the first device can update the device table. The first device can determine whether the traversal direction has changed based on the change in the coordinate direction. If the traversal direction has changed, the first device can update the device table.

[0103] The keyboard and mouse traversal method provided in this application can update device information in real time when the detected device angle changes, which is conducive to accurately understanding the device information and improving the accuracy of keyboard and mouse traversal.

[0104] In combination with the first aspect, in certain implementations of the first aspect, the coordinate directions include: left, right, and center, the first direction is left or right, and the first device determines the coordinate direction after movement and the crossing direction after movement based on the device angle after movement. The method also includes: when the coordinate direction after movement is a second direction opposite to the first direction, if there is a third device connected in the second direction, the crossing direction after movement is the same as the crossing direction before movement; when the coordinate direction after movement is a second direction opposite to the first direction, if there is no third device connected in the second direction, the crossing direction after movement is opposite to the crossing direction before movement; when the coordinate direction after movement is center, the crossing direction after movement is the default crossing direction or the same as the crossing direction before movement.

[0105] The crossing direction between the first device and the second device is related to the coordinate direction of the first device relative to the second device.

[0106] If the coordinate direction of the second device relative to the first device changes, and the coordinate direction before movement (i.e., the first direction) is left or right, and the coordinate direction after movement is opposite to the coordinate direction before movement, the first device detects whether there is a connected third device in the coordinate direction after movement. If so, the crossing direction after movement is the same as the crossing direction before movement; if not, the crossing direction after movement is opposite to the crossing direction before movement.

[0107] If the coordinate direction of the second device relative to the first device changes, and the coordinate direction before movement (i.e., the first direction) is left or right, and the coordinate direction after movement is center, the crossing direction after movement is the default crossing direction or the same as the crossing direction before movement (also called the historical crossing direction).

[0108] In combination with the first aspect, in some implementations of the first aspect, the cursor is displayed as a first shape on the first screen, and the cursor is displayed as a second shape on the second screen, and the first shape and the second shape are different.

[0109] The cursor shape can be different when it travels between different devices.

[0110] In the example shown in FIG2 , the shape of the cursor displayed on the computer screen is an arrow, and the shape of the cursor displayed on the mobile phone screen is a circle.

[0111] In combination with the first aspect, in some implementations of the first aspect, the first device is a Windows system, and the second device is an Android system.

[0112] In a second aspect, a terminal device, also referred to as a first device, is provided. The first device is connected to an input device for inputting control instructions, and includes a first screen; the first device includes a display module and a processing module. The display module is configured to: display a cursor corresponding to the input device on the first screen; the processing module is configured to: detect a first operation applied to the input device, the first operation being to drag the cursor to determine a first direction as a connection direction; in response to the first operation, establish a connection with a second device located in the first direction of the first device; detect a second operation applied to the input device, the second operation being to drag the cursor displayed on the first screen in the first direction; in response to the second operation, control the cursor to move out of the first screen and onto the second screen of the second device for display; and in response to a third operation applied to the input device, the second device executes a control instruction corresponding to the third operation, the third operation being to input a control instruction on the second device.

[0113] In combination with the second aspect, in some implementations of the second aspect, the processing module is further configured to: in response to the first operation, find a second device that meets the connection condition in the first direction; and establish a connection with the second device.

[0114] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned processing module is also used to: perform device discovery; store the device identification of the device that meets the connection conditions and the direction of the device that meets the connection conditions relative to the first device in a device table; and in response to the first operation, find a second device that meets the connection conditions in the first direction through the device table.

[0115] In combination with the second aspect, in some implementations of the second aspect, the processing module is further used to: detect a change in the direction of the second device relative to the first device; and update the direction of the second device relative to the first device in the device table.

[0116] In combination with the second aspect, in some implementations of the second aspect, the processing module is further used to: detect a fourth operation of the user, where the fourth operation is used to select the second device.

[0117] In combination with the second aspect, in some implementations of the second aspect, the above-mentioned display module is also used to: display a first interface, the first interface includes an automatic connection option; the above-mentioned processing module is also used to: respond to an operation acting on the automatic connection option, and the automatic connection option is selected.

[0118] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned display module is also used to: display a second interface in response to an operation of opening a first application on a first device, the second interface including: a first icon for representing the first device, a second icon for representing the second device, and placement areas respectively located on both sides of the first icon; the above-mentioned processing module is also used to: establish a connection with the second device in response to a fifth operation of dragging the second icon to the first placement area, the first placement area being in the first direction of the first device.

[0119] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned processing module is also used to: in response to the fifth operation, determine whether there is a connected device in the first direction; if there is no connected device, establish a connection with the second device; if there is a connected device, prohibit connection with the second device, and restore the position of the second icon to the position before dragging.

[0120] In combination with the second aspect, in some implementations of the second aspect, the connection condition is that the login account of the second device is the same as that of the first device.

[0121] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned processing module is also used to: in response to the second operation, obtain the exit position of the cursor moving out of the first screen; based on the exit position, determine the initial position of the cursor entering the second screen; and send the initial position to the second device to instruct the second device to display the cursor at the initial position of the second screen.

[0122] In combination with the second aspect, in some implementations of the second aspect, the processing module is further configured to determine an initial position based on the exit position, the screen resolution of the first screen, and the screen resolution of the second screen.

[0123] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned processing module is also used to: determine the first conversion rate based on the screen resolution of the first screen and the screen resolution of the second screen; and determine the initial position based on the exit position and the first conversion rate.

[0124] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned processing module is also used to: obtain a first movement speed of the cursor on the first screen in response to a second operation; determine a second movement speed of the cursor on the second screen based on the first movement speed; and send the second movement speed to the second device to instruct the second device to move the cursor at the second movement speed on the second screen.

[0125] In combination with the second aspect, in some implementations of the second aspect, the processing module is further used to: determine the second moving speed based on the first moving speed, the screen resolution of the first screen, and the screen resolution of the second screen.

[0126] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned processing module is also used to: determine the second conversion rate based on the screen resolution of the first screen and the screen resolution of the second screen; determine the second moving speed based on the first moving speed and the second conversion rate.

[0127] In conjunction with the second aspect, in certain implementations of the second aspect, the first device further includes a transceiver module. The transceiver module is configured to receive a third operation; the processing module is further configured to intercept an input event corresponding to the third operation; and transmit the input event to the second device, wherein the second device determines a control instruction corresponding to the third operation based on the input event.

[0128] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned processing module is also used to: detect a sixth operation acting on the input device, the sixth operation being used to drag the cursor displayed on the second screen toward the first device; in response to the sixth operation, control the cursor to move out of the second screen and enter the first screen of the second device for display; stop intercepting input events corresponding to the operation acting on the input device to execute control instructions corresponding to the operation of the input device.

[0129] In conjunction with the second aspect, in certain implementations of the second aspect, the first device further includes a transceiver module. The transceiver module is configured to: receive a seventh operation applied to the input device, the seventh operation being used to input a control instruction for forcibly reclaiming the cursor on the second device; the processing module is further configured to: control the cursor to be displayed on the first screen in response to the seventh operation; and stop intercepting input events corresponding to the operation applied to the input device to execute the control instruction corresponding to the operation of the input device.

[0130] In combination with the second aspect, in certain implementations of the second aspect, the first direction is left or right, and the above-mentioned processing module is used to: detect that the second device moves from the first direction to a second direction opposite to the first direction; in response to an eighth operation, control the cursor to move out of the first screen and enter the second screen of the second device for display, and the eighth operation is opposite to the direction of dragging the cursor in the second operation.

[0131] In combination with the second aspect, in certain implementations of the second aspect, the first direction is left or right, and the above-mentioned processing module is used to: detect that the second device moves from the first direction to the middle; in response to the ninth operation, control the cursor to move out of the first screen and enter the second screen of the second device for display, and the ninth operation is the same as the direction of dragging the cursor in the second operation.

[0132] In combination with the second aspect, in certain implementations of the second aspect, the first direction is center, and the above-mentioned processing module is used to: detect that the second device moves from the first direction to the second direction, and the second direction is left or right; in response to the tenth operation, the first device controls the cursor to move out of the first screen and enter the second screen of the second device for display, and the tenth operation is the same as the second direction.

[0133] In combination with the second aspect, in certain implementations of the second aspect, the first direction is left or right, and the first device is connected to a third device located in the second direction of the first device, and the second direction is opposite to the first direction; the above-mentioned processing module is used to: detect that the second device moves from the first direction to the second direction; in response to the eleventh operation, the first device controls the cursor to move out of the first screen and enter the second screen of the second device for display, and the eleventh operation is the same as the direction of dragging the cursor in the second operation.

[0134] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned processing module is also used to: discover the second device; obtain the device identification of the second device and the device angle of the second device relative to the first device; determine the coordinate direction and crossing direction of the second device based on the device angle; and record the device identification, device angle, coordinate direction and crossing direction in the device table.

[0135] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned processing module is also used to: detect the movement of the second device; when the second device is detected to be stationary, continue to detect the device angle of the second device after the movement; if the device angle after the movement and the device angle before the movement do not belong to the same angle range, determine the coordinate direction after the movement and the crossing direction after the movement according to the device angle after the movement, wherein each coordinate direction corresponds to a preset angle range; and update the device table based on the coordinate direction after the movement and the crossing direction after the movement.

[0136] In combination with the second aspect, in certain implementations of the second aspect, the coordinate directions include: left, right, and center, the first direction is left or right, and the above-mentioned processing module is also used to: when the coordinate direction after movement is a second direction opposite to the first direction, if there is a third device connected in the second direction, the crossing direction after movement is the same as the crossing direction before movement; when the coordinate direction after movement is a second direction opposite to the first direction, if there is no third device connected in the second direction, the crossing direction after movement is opposite to the crossing direction before movement; when the coordinate direction after movement is center, the crossing direction after movement is the default crossing direction or the same as the crossing direction before movement.

[0137] In combination with the second aspect, in some implementations of the second aspect, the cursor is displayed as a first shape on the first screen, and the cursor is displayed as a second shape on the second screen, and the first shape and the second shape are different.

[0138] In combination with the second aspect, in some implementations of the second aspect, the first device is a Windows system and the second device is an Android system.

[0139] In a third aspect, the present application provides a terminal device, comprising a processor coupled to a memory and configured to execute instructions in the memory to implement the method of any possible implementation of the first aspect. Optionally, the terminal device further comprises a memory. Optionally, the terminal device further comprises a transceiver, the processor coupled to the transceiver.

[0140] In a fourth aspect, the present application provides a processor comprising: an input circuit, an output circuit, and a processing circuit. The processing circuit is configured to receive a signal through the input circuit and transmit a signal through the output circuit, so that the processor executes the method of any possible implementation of the first aspect.

[0141] In a specific implementation, the processor may be a chip, the input circuit may be an input pin, the output circuit may be an output pin, and the processing circuit may be a transistor, a gate circuit, a trigger, or various logic circuits. The input signal received by the input circuit may be, for example, but not limited to, received and input by a receiver, and the signal output by the output circuit may be, for example, but not limited to, output to and transmitted by a transmitter. The input circuit and the output circuit may be the same circuit, which functions as an input circuit and an output circuit at different times. This application does not limit the specific implementation of the processor and various circuits.

[0142] In a fifth aspect, the present application provides a processing device comprising a processor and a memory. The processor is configured to read instructions stored in the memory and to receive signals via a receiver and transmit signals via a transmitter to execute the method in any possible implementation of the first aspect.

[0143] Optionally, there are one or more processors and one or more memories.

[0144] Optionally, the memory may be integrated with the processor, or the memory may be provided separately from the processor.

[0145] In the specific implementation process, the memory can be a non-transitory memory, such as a read-only memory (ROM), which can be integrated on the same chip as the processor or be set on different chips. This application does not limit the type of memory and the setting method of the memory and the processor.

[0146] It should be understood that related data interaction processes, such as sending indication information, can be the process of outputting indication information from the processor, and receiving capability information can be the process of receiving input capability information from the processor. Specifically, the output data of the processing can be output to the transmitter, and the input data received by the processor can come from the receiver. The transmitter and receiver can be collectively referred to as a transceiver.

[0147] The processing device in the seventh aspect mentioned above can be a chip. The processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading the software code stored in the memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0148] In an eighth aspect, the present application provides a computer-readable storage medium, which stores a computer program (also referred to as code, or instructions) which, when run on a computer, enables the computer to execute the method in any possible implementation of the first aspect above.

[0149] In a ninth aspect, the present application provides a computer program product, which includes: a computer program (also referred to as code, or instructions), which, when executed, enables a computer to execute a method in any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0150] FIG1 is a schematic diagram of a communication system applicable to an embodiment of the present application;

[0151] FIG2 is a schematic diagram of a keyboard and mouse traversal system provided in an embodiment of the present application;

[0152] FIG3 is a schematic diagram of a keyboard and mouse forced recapture method provided by an embodiment of the present application;

[0153] FIG4 is a schematic diagram of a cross-device photo sharing method provided by an embodiment of the present application;

[0154] FIG5 is a schematic diagram of an interface of a computer manager provided in an embodiment of the present application;

[0155] FIG6 is a schematic diagram of another keyboard and mouse traversal method provided in an embodiment of the present application;

[0156] FIG7 is a schematic diagram of another keyboard and mouse traversal method provided in an embodiment of the present application;

[0157] FIG8 is a schematic diagram of another keyboard and mouse traversal method provided in an embodiment of the present application;

[0158] FIG9 is a schematic diagram of a visualization interface provided in an embodiment of the present application;

[0159] FIG10 is a schematic diagram of a position change provided by an embodiment of the present application;

[0160] FIG11 is a schematic diagram of a position provided in an embodiment of the present application;

[0161] FIG12 is a schematic flow chart of a keyboard and mouse traversal control method provided in an embodiment of the present application;

[0162] FIG13 is a block diagram of a software structure of a PC provided in an embodiment of the present application;

[0163] FIG14 is a software structure block diagram of an Android device provided in an embodiment of the present application;

[0164] FIG15 is a schematic flow chart of a startup method in a keyboard and mouse scenario provided by an embodiment of the present application;

[0165] FIG16 is a schematic flow chart of an automatic connection method provided in an embodiment of the present application;

[0166] FIG17 is a schematic flow chart of a manual connection method provided in an embodiment of the present application;

[0167] FIG18 is a schematic flow chart of a keyboard and mouse connection method provided in an embodiment of the present application;

[0168] FIG19 is a schematic diagram of a method for detecting a mouse position provided in an embodiment of the present application;

[0169] FIG20 is a schematic diagram of establishing a screen coordinate system provided by an embodiment of the present application;

[0170] FIG21 is a schematic flow chart of a keyboard and mouse traversal method provided in an embodiment of the present application;

[0171] FIG22 is a schematic flow chart of a method for determining an initial position of a mouse traversal provided in an embodiment of the present application;

[0172] FIG23 is a schematic flow chart of a method for determining mouse movement speed provided in an embodiment of the present application;

[0173] FIG24 is a schematic flow chart of a method for intercepting keyboard and mouse events on a PC provided in an embodiment of the present application;

[0174] FIG25 is a schematic flowchart of a method for forcibly reclaiming a keyboard and mouse provided in an embodiment of the present application;

[0175] FIG26 is a schematic flowchart of a method for adjusting a crossing direction provided in an embodiment of the present application;

[0176] FIG27 is a schematic flowchart of another method for adjusting the crossing direction provided by an embodiment of the present application;

[0177] FIG28 is a schematic block diagram of a terminal device provided in an embodiment of the present application;

[0178] Figure 29 is a schematic block diagram of another terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0179] The technical solution in this application will be described below with reference to the accompanying drawings.

[0180] To facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or execution order, and the words "first" and "second" do not necessarily mean different.

[0181] It should be noted that, in this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0182] In addition, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b and c can mean: a, or b, or c, or a and b, or a and c, or b and c, or a, b and c, where a, b, c can be single or multiple.

[0183] The embodiment of the present application provides a keyboard and mouse traversal method, which can operate different terminal devices through the mouse and keyboard to achieve seamless access between different terminal devices.

[0184] In order to better understand the embodiments of the present application, the communication system to which the embodiments of the present application are applicable is first introduced.

[0185] Figure 1 shows a schematic diagram of a communication system 100. As shown in Figure 1, the communication system 100 includes a computer 101, a mobile phone 102, and a tablet 103. Among them, the operating system of the computer 101 is the Windows system, and the operating systems of the mobile phone 102 and the tablet 103 are the Android system. Keyboard and mouse can be passed between the computer 101 and the mobile phone 102, and keyboard and mouse can be passed between the computer 101 and the tablet 103. In other words, the computer 101, the mobile phone 102, and the tablet 103 can share a mouse and keyboard. In this case, the computer 101 can be called a source device or a sending end, and the mobile phone 102 and the tablet 103 can be called a peer device or a receiving end, but the embodiments of the present application are not limited to this.

[0186] Computer 101 can achieve keyboard and mouse pass-through with two devices. Both devices can be Android systems. In the example shown in Figure 1, computer 101 can achieve keyboard and mouse pass-through with mobile phone 102 and tablet 103. Alternatively, computer 101 can achieve keyboard and mouse pass-through only with mobile phone 102 or tablet 103. In other words, computer 101 and mobile phone 102 share the same mouse and keyboard, or computer 101 and tablet 103 share the same mouse and keyboard. In this case, computer 101 is the source device, and mobile phone 102 or tablet 103 is the peer device.

[0187] The left or right side of the computer 101 can achieve keyboard and mouse crossover with a device, that is, the computer 101 can support the mouse crossover from the left side of the computer 101 screen to another device, and can also support the mouse crossover from the right side of the computer 101 screen to another device. If the computer 101 can achieve keyboard and mouse crossover with the mobile phone 102 and tablet 103, then the mobile phone 102 is on the right side of the computer 101 and the tablet 103 is on the left side of the computer 101, or the mobile phone 102 is on the left side of the computer 101 and the tablet 103 is on the right side of the computer 101. If both the mobile phone 102 and the tablet 103 are on the left side of the computer 101, or both the mobile phone 102 and the tablet 103 are on the right side of the computer 101, the computer 101 can achieve keyboard and mouse crossover with one device (mobile phone 102 or tablet 103) at a time.

[0188] After the keyboard and mouse of the computer 101 are transferred to the mobile phone 102 and / or tablet 103, the user can use the mouse to click, drag or double-click the mobile phone 102 and / or tablet 103, and can use the keyboard to input information on the mobile phone 102 and / or tablet 103.

[0189] For example, Figure 2 shows a schematic diagram of keyboard and mouse traversal. As shown in interface a in Figure 2, the computer 101 can achieve keyboard and mouse traversal with the mobile phone 102, that is, the mouse pointer 1011 displayed on the computer 101 can traverse from the screen of the computer 101 to the screen of the mobile phone 102. When the computer 101 detects the traversal operation of the mouse pointer 1011, the b interface in Figure 2 can be displayed. As shown in interface b in Figure 2, the mouse pointer 1011 can be displayed in the form of a circle on the screen of the mobile phone 102. The user can click on the memo icon with the mouse pointer 1011, and the mobile phone 102 can display interface c in Figure 2 in response to the user's operation of clicking the memo icon. As shown in interface c in Figure 2, the mobile phone 102 displays the memo interface. The user can enter the information "Unprocessed matters" on the memo interface through the keyboard, and the mobile phone 102 displays the text "Unprocessed matters" on the memo interface in response to the user's operation of entering information on the memo interface.

[0190] It should be noted that in this example, the mouse pointer 1011 passes from the computer 101 to the mobile phone 102 and is displayed as a circle on the mobile phone 102. When the mobile phone 102 is connected to an external mouse and / or keyboard, the mobile phone 102 also displays the mouse pointer in the form of a circle. In other examples, the mouse pointer may also be displayed in other forms after passing from the computer to the mobile phone, and this is not limited here.

[0191] After the keyboard and mouse of computer 101 are transferred to mobile phone 102 and / or tablet 103, the user can forcibly reclaim the keyboard and mouse using a shortcut key. When computer 101 detects that the user has triggered the shortcut key for forcibly reclaiming the keyboard and mouse, the mouse can be displayed in a designated position on the screen of computer 101 in response to the user's operation. The shortcut key for forcibly reclaiming the keyboard and mouse can be Esc or Ctrl+Alt+DELETE, and the designated position on the screen can be the center of the screen, but embodiments of the present application are not limited thereto.

[0192] For example, Figure 3 shows a schematic diagram of forced keyboard and mouse recapture. As shown in Figure 3, computer 101 can achieve keyboard and mouse interoperability with mobile phone 102. A mouse pointer 1011 is displayed on the screen of mobile phone 102. Computer 101 detects the user's Esc key press and, in response, displays mouse pointer 1011 in the center of computer 101's screen.

[0193] After the keyboard and mouse of the computer 101 are transferred to the mobile phone 102 and / or tablet 103, files, photos, music and other content can also be transferred across devices. In other words, the user can drag and drop files, photos, music and other content from the mobile phone 102 and / or tablet 103 to the computer 101, and can also drag and drop files, photos, music and other content from the computer 101 to the mobile phone 102 and / or tablet 103. It is understood that the embodiments of the present application are not limited to content such as files, photos and music.

[0194] For example, FIG4 shows a schematic diagram of sharing photos across devices. As shown in interface a of FIG4 , the computer 101 can realize keyboard and mouse crossover with the mobile phone 102. When the user takes a screenshot of the main interface, the mobile phone 102 takes a screenshot in response to the user's screenshot operation and displays the screenshot image in the lower left corner of the interface. The user drags the screenshot image to the desktop of the computer 101 using the mouse. When the computer 101 detects that the user has dragged the image to the main interface of the computer 101, it can display interface b of FIG4 in response to the operation. As shown in interface b of FIG4 , the computer 101 can display the drag icon of the screenshot image, and accordingly, the screenshot image in the lower right corner of the interface of the mobile phone 102 disappears. When the computer 101 detects that the user has canceled the drag operation on the screenshot image, it can save the screenshot image to the file manager in response to the operation and display the interface of the file manager, as shown in interface c of FIG4 . In the c interface in FIG4 , the computer 101 saves the screenshot image to the file manager, and names the screenshot image as 20220405_1 by default, displays the format of the screenshot image as png, displays the size of the screenshot image as 302.08 kilobytes (kb), and displays the time of saving the screenshot image as “just now”.

[0195] It should be noted that the file manager can support any type of content, so the computer 101 can be used to save content such as pictures, music, and files dragged to the main interface.

[0196] Alternatively, if the user drags the screenshot image to a gallery application in the computer 101, the computer 101 can save the screenshot image to the gallery application in response to the user's operation. It is understood that if the user drags a file to the gallery application in the computer 101, and the gallery application does not support the file format, the computer 101 cannot save the file to the gallery application.

[0197] The computer 101 can achieve keyboard and mouse cross-connection with the mobile phone 102 and the tablet 103 if the computer 101, the mobile phone 102 and the tablet 103 are logged in using the same account. When the computer 101, the mobile phone 102 and the tablet 103 are logged in using the same account, the computer 101, the mobile phone 102 and the tablet 103 can be connected via WiFi or Bluetooth, where WiFi can include a local area network or peer-to-peer (P2P) communication.

[0198] Computer 101 can sense all devices that meet the requirements for keyboard and mouse pass-through. However, the number of devices that can achieve keyboard and mouse pass-through with computer 101 can be two. When computer 101 senses more than two devices, computer 101 can achieve keyboard and mouse pass-through with the device selected by the user based on the user's operation to select the device. Alternatively, computer 101 can independently select the device for keyboard and mouse pass-through based on preset rules. This embodiment of the application does not limit the preset rules.

[0199] Computer 101 is connected to mobile phone 102 and tablet 103 to realize keyboard and mouse crossing, including two implementation methods. One possible implementation method is manual. The automatic selection option is an option for computer 101 to independently select the connection device. When the automatic selection option is turned off (or not turned on), the user can manually drag or click the connectable device in the visual interface to connect computer 101 to mobile phone 102 and tablet 103. Another possible implementation method is automatic. When the automatic selection option is turned on and mobile phone 102 and tablet 103 meet the keyboard and mouse crossing conditions, computer 101 can automatically connect to mobile phone 102 and tablet 103. Among them, the automatic selection option can be in different positions in different devices. For example, the automatic selection option can be in the computer manager of computer 101. The automatic selection option can be in the control center or settings of mobile phone 102 and tablet 103.

[0200] For example, Figure 5 shows a schematic diagram of the interface of a computer manager. As shown in Figure 5, the computer manager interface may include options such as homepage, multi-screen collaboration, hyperterminal, online tips, official services, smart audition, and system optimization. When the computer 101 detects that the user clicks the hyperterminal option with the mouse, the super keyboard and mouse interface can be displayed. The super keyboard and mouse interface includes two options. Among them, the first option is "Touch the mouse pointer twice in a row on the side edge of the computer screen to establish a connection." The second option is "Automatically adjust the screen arrangement as the device placement changes." In the current example, the first option is selected. When the computer 101 detects that the mouse pointer touches the side edge of the computer screen twice in a row, the computer 101 can connect to the device that meets the keyboard and mouse cross-conditions. It can be understood that the first option is the above-mentioned automatic connection option. When this option is selected, the computer 101 can automatically connect to the device that meets the keyboard and mouse cross-conditions. When the first option (i.e., the automatic connection option) is not selected, the computer 101 provides a manual method to connect to the device that meets the keyboard and mouse cross-conditions.

[0201] The Super Keyboard and Mouse interface in Figure 5 also displays connected and online devices. As shown, a tablet is connected to the left side of the PC, along with three online devices: a Magi4, a Magi4xy, and a Magi4y. If the automatic connection option is disabled, users can select a device from these three online devices by clicking or dragging.

[0202] If the way in which the computer 101, the mobile phone 102 and the tablet 103 realize keyboard and mouse crossing is automatic, when the computer 101 detects that the mouse is at the edge of the screen, a bubble box can be displayed at the edge of the screen to indicate the presence of a traversable device. Among them, the bubble box is only an example of a name, and the embodiment of the present application does not limit this. The number of traversable devices can be one or more, and the embodiment of the present application does not limit this. When the number of traversable devices on one side is one, the number of bubble boxes can be one. The bubble box can display the icon and model of the traversable device, for example: if the traversable device is a mobile phone, the mobile phone icon and the model of the mobile phone can be displayed: mobile phone Magic4, or the icon and / or model of the traversable device may not be displayed, and the embodiment of the present application does not limit this.

[0203] When there are multiple traversable devices, the number of bubble boxes can be one, or the same as the number of traversable devices, and this embodiment of the present application does not limit this. If there are multiple traversable devices, the number of bubble boxes is multiple, each of the multiple bubble boxes can correspond to a traversable device, and each bubble box can display the icon and / or model of the corresponding traversable device. The user can click on the bubble box to select a traversable device from multiple traversable devices for keyboard and mouse traversal, or can select a traversable device from multiple traversable devices for keyboard and mouse traversal by clicking on the icon or model of the traversable device. This embodiment of the present application does not limit this. If there are multiple traversable devices, the number of bubble boxes is one, and this bubble box can display the icons and / or models of multiple traversable devices. The user can select a traversable device from multiple traversable devices for keyboard and mouse traversal by clicking on the icon or model of the traversable device.

[0204] For example, Figure 6 shows a schematic diagram of keyboard and mouse traversal. As shown in Figure 6, tablet 103 is to the right of computer 101. Computer 101 has one traversable device and one bubble box, and the bubble box does not display the icon and / or model of the traversable device. When computer 101 detects that the mouse is at the right edge of the screen, bubble box 1012 is displayed at the right edge of the screen to indicate the presence of a traversable device. The user can slide the mouse to the right to reach the screen of tablet 103.

[0205] For example, Figure 7 shows a schematic diagram of another keyboard and mouse traversal. As shown in Figure 7, the mobile phone 102 and the tablet 103 are both on the right side of the computer 101. The number of traversable devices of the computer 101 is two, the number of bubble boxes is one, and the bubble box displays the icons of the two traversable devices. When the computer 101 detects that the mouse is at the right edge of the screen, a bubble box 1012 is displayed on the right edge of the screen to indicate the presence of a traversable device, and the bubble box displays the icon of the mobile phone 102 and the icon of the tablet 103. The user can click on the icon 10121 of the mobile phone 102 with the mouse pointer to select the mobile phone 102, slide the mouse to the right, and the mouse pointer can reach the screen of the mobile phone 102. The user can also click on the icon 10122 of the tablet 103 with the mouse pointer to select the tablet 103, slide the mouse to the right, and the mouse can reach the screen of the tablet 103.

[0206] For example, FIG8 shows another schematic diagram of keyboard and mouse crossing. As shown in FIG8 , the mobile phone 102 and the tablet 103 are both on the right side of the computer 101. The number of traversable devices of the computer 101 is two, the number of bubble boxes is two, and each bubble box displays an icon of a traversable device. When the computer 101 detects that the mouse is at the right edge of the screen, a bubble box 1013 is displayed on the right edge of the screen to indicate the presence of the traversable device mobile phone 102 and a bubble box 1014 is displayed to indicate the presence of the traversable device tablet 103. The bubble box 1013 displays the icon 10131 of the mobile phone 102, and the bubble box 1014 displays the icon 10141 of the tablet 103. The user can click on the icon 10131 of the mobile phone 102 or the bubble box 1013 with the mouse pointer to select the mobile phone 102, and slide the mouse to the right so that the mouse pointer can reach the screen of the mobile phone 102. The user can also use the mouse pointer to click the icon 10141 or the bubble box 1014 of the tablet 103 to select the tablet 103, and slide the mouse to the right so that the mouse can reach the screen of the tablet 103.

[0207] When computer 101 implements keyboard and mouse panning with mobile phone 102 or tablet 103, if the position of mobile phone 102 and / or tablet 103 relative to 101 changes, computer 101 can identify the position of mobile phone 102 and tablet 103 relative to computer 101 and use this position to determine the panning direction of the mouse. In the example shown in Figure 1, if mobile phone 102 is to the right of computer 101, the panning direction of the mouse is right panning. If tablet 103 is to the left of computer 101, the panning direction of the mouse is left panning.

[0208] The computer 101 can determine the position of the mobile phone 102 and / or tablet 103 relative to the computer 101 through two modes, thereby determining the mouse's traversal direction, wherein the two modes are manual mode and smart mode.

[0209] In one possible implementation, the user can set the computer 101 to manually adjust the position of the mobile phone 102 and / or tablet 103 in the PC Manager of the computer 101. In this implementation, the user can manually adjust the position of the mobile phone 102 and tablet 103 through a visual interface. If the user manually adjusts the position of the mobile phone 102 and tablet 103 through the visual interface, the computer 101 updates the mouse traversal direction in response to the user's adjustment operation.

[0210] For example, the computer 101 can support the user to swap the positions of the mobile phone 102 and the tablet 103 in the visual interface. If the computer 101 only realizes keyboard and mouse crossover with the mobile phone 102 or tablet 103, the computer 101 can support the user to adjust the mobile phone 102 or tablet 103 from the right side of the computer 101 to the left side, or adjust the mobile phone 102 or tablet 103 from the left side of the computer 101 to the right side in the visual interface.

[0211] The above visual interface can be found in PC Manager.

[0212] For example, Figure 9 shows a schematic diagram of a visualization interface. As shown in Figure 9, the Super Keyboard and Mouse interface may include a visualization area 1015. Visualization area 1015 includes a computer, a tablet, and perceptible devices that meet the keyboard and mouse crossover requirements. The computer and tablet can achieve keyboard and mouse crossover, with the tablet on the left side of the computer. The user can adjust the position of the tablet within visualization area 1015, for example, from the left side of the computer to the right side.

[0213] In one example, if keyboard and mouse crossover is implemented between computer 101, mobile phone 102, and tablet 103, computer 101 does not support the user adjusting mobile phone 102 and tablet 103 to the same side. If the user adjusts mobile phone 102 and tablet 103 to the same side for operation, computer 101 detects the user's operation and restricts the operation, that is, the original position is still displayed.

[0214] For example, in the example shown in Figure 9 above, the tablet is on the left side of the computer. If the user manually drags the device that meets the keyboard and mouse crossing conditions to the left side of the computer, the computer 101 detects the user's operation and restores the device that meets the keyboard and mouse crossing conditions to its original position.

[0215] In another possible implementation, when the user turns on the option of automatically identifying the position, the computer 101 can detect the position change of the mobile phone 102 or the tablet 103 through ultrasonic method.

[0216] For example, in the interface shown in FIG9 , if the user checks or selects the option “Automatically adjust screen arrangement as device placement changes,” the computer 101 detects the user's check or selection operation and, in response to the operation, can automatically identify the position change of the mobile phone 102 or tablet 103 and adjust the screen arrangement of different devices based on the changed position. The mobile phone 102 or tablet 103 can detect its own movement through a motion sensor. When the movement stops, the position of the computer 101 relative to the mobile phone 102 or tablet 103 is calculated using data obtained by the ultrasonic sensor and the position is transmitted to the computer 101. The computer 101 receives the position and can determine the position of the mobile phone 102 or tablet 103 relative to the computer 101.

[0217] Figure 10 shows a schematic diagram of a position change. As shown in Figure 10, the mobile phone 102 is on the left side of the computer 101. When the user moves the mobile phone 102 to the right side of the computer 101, the computer 101 can detect that the mobile phone 102 is on the right side of the computer 101 through ultrasonic method.

[0218] The computer 101 may update the mouse traversal direction according to the position of the mobile phone 102 and / or tablet 103 according to the following strategy:

[0219] 1) When the computer 101 realizes keyboard and mouse crossover with the mobile phone 102 and the tablet 103, the mobile phone 102 and the tablet 103 are located on both sides (left and right) of the computer 101. If the position of the mobile phone 102 changes and changes to the same side as the tablet 103, or the position of the tablet 103 changes and changes to the same side as the mobile phone 102, the computer 101 does not change the original mouse crossover direction.

[0220] For example, in the example shown in FIG1 , mobile phone 102 is on the right side of computer 101, and the mouse movement direction is right-travel. Tablet 103 is on the left side of computer 101, and the mouse movement direction is left-travel. If computer 101 detects that mobile phone 102 has moved from the right side of computer 101 to the left side of computer 101 without tablet 103 changing position, the mouse movement directions of both mobile phone 102 and tablet 103 remain unchanged: that is, the mouse movement direction of mobile phone 102 remains right-travel, and the mouse movement direction of tablet 103 remains left-travel.

[0221] 2) When the computer 101 realizes keyboard and mouse crossover with the mobile phone 102 and the tablet 103, if it is detected that the positions of the mobile phone 102 and the tablet 103 are swapped, the computer 101 swaps the keyboard and mouse crossover directions of the mobile phone 102 and the tablet 103.

[0222] For example, in the example shown in FIG1 , mobile phone 102 is on the right side of computer 101, and the mouse's panning direction is right panning. Tablet 103 is on the left side of computer 101, and the mouse's panning direction is left panning. If computer 101 detects that mobile phone 102 is on the left side of computer 101 and tablet 103 is on the right side of computer 101, the mouse's panning direction for mobile phone 102 is updated to left panning, and the mouse's panning direction for tablet 103 is updated to right panning.

[0223] 3) When computer 101 and mobile phone 102 or tablet 103 implement keyboard and mouse crossover, if mobile phone 102 or tablet 103 is on the right side of computer 101 and detects that mobile phone 102 or tablet 103 has moved to the left side of computer 101, computer 101 updates the crossover direction of mobile phone 102 or tablet 103's mouse to left crossover. Alternatively, if mobile phone 102 or tablet 103 is on the left side of computer 101 and detects that mobile phone 102 or tablet 103 has moved to the right side of computer 101, computer 101 updates the crossover direction of mobile phone 102 or tablet 103's mouse to right crossover.

[0224] For example, in the example shown in FIG. 9 , if the user adjusts the tablet from the left side of the computer to the right side of the computer, the computer 101 can change the direction of the computer-to-tablet crossing to right crossing in response to the user's operation.

[0225] For example, in the example shown in FIG10 , the computer 101 detects that the mobile phone 102 is adjusted from the left side of the computer to the right side of the computer 101 , and the crossing direction from the computer 101 to the mobile phone 102 can be updated to right crossing.

[0226] It should be noted that the position of the mobile phone 102 and / or tablet 103 relative to the computer 101 can be on the left, right, or center. If the mobile phone 102 and / or tablet 103 is on the left or right side relative to the computer 101, the mouse's panning direction can be determined according to the above rules. If the mobile phone 102 and tablet 103 are in the center relative to the computer 101, the mouse's panning direction remains unchanged.

[0227] The computer 101 can establish a coordinate system with the center of the screen as the origin, parallel lines of the long side of the screen, and parallel lines of the wide side of the screen, so as to divide the left area, the right area, and the middle area. If the computer 101 detects that the mobile phone 102 and / or the tablet 103 is in the left area, it can be determined that the mobile phone 102 and / or the tablet 103 is on the left side of the computer 101. If the computer 101 detects that the mobile phone 102 and / or the tablet 103 is in the right area, it can be determined that the mobile phone 102 and / or the tablet 103 is on the right side of the computer 101. If the computer 101 detects that the mobile phone 102 and / or the tablet 103 is in the middle area, it can be determined that the mobile phone 102 and / or the tablet 103 is in the middle of the computer 101.

[0228] For example, Figure 11 shows a schematic diagram of the positions. As shown in Figure 11, the computer 101 sets the area between (-60°, 60°) in the coordinate system as the right area, the area between (60°, 120°) and (-60°, -120°) as the middle area, and the area between (-120°, 120°) as the left area. If the angle of the mobile phone 102 relative to the computer 101 is between (-60°, 60°), the computer 101 detects that the mobile phone 102 is in the right area and can determine that the mobile phone 102 is on the right side of the computer 101.

[0229] It should be noted that the angle range of the left area and the right area shown in Figure 9 based on the middle area is only an example. The angle range of the middle area can be adjusted according to the user's experience. For example, the angle range of the middle area can be adjusted from (60°, 120°) to (30°, 150°) and from (-60°, -120°) to (-30°, -150°) according to the user's experience.

[0230] In summary, the functions that can be supported by the computer 101 provided in the embodiment of the present application are shown in Table 1.

[0231] As shown in Table 1, when the number of devices is 2, the computer can be connected to 1 mobile phone or 1 tablet. When the number of devices is 3, the computer can be connected to 1 mobile phone and 1 tablet, or the computer can be connected to 2 mobile phones, or the computer can be connected to 2 tablets. It can be understood that the computer can be connected to a mobile phone and / or tablet, which means that the computer can realize keyboard and mouse crossing with the mobile phone and / or tablet. The condition for the computer to connect to the mobile phone and / or tablet is: automatic connection under the same account. The management entrance for the computer to connect to other devices (mobile phones and / or tablets) is the computer manager, and the management entrance for the mobile phone and tablet to connect to other devices (computers) is the control center and settings. The connection control methods of the computer and other devices include manual and automatic. The computer can be controlled in direction by manual setting or automatic recognition.

[0232] In order to realize the functions of Table 1, an embodiment of the present application provides a method for controlling keyboard and mouse traversal.

[0233] For example, FIG12 shows a schematic flow chart of a method 1200 for controlling keyboard and mouse traversal. The method can be executed by the computer 101. As shown in FIG12, the method 1200 includes the following steps:

[0234] S1201: Detect device E that meets the keyboard and mouse crossing conditions.

[0235] The number of devices E can be one or more, which is not limited in the present embodiment. The device E with keyboard and mouse traversal condition can also be called a traversable device, which is not limited in the present embodiment.

[0236] The device E may be the mobile phone 102 and / or tablet 103 described above, but the embodiment of the present application is not limited thereto.

[0237] S1202: Determine whether the automatic connection option is enabled.

[0238] The automatic connection option can be as shown in Figure 5. If the automatic connection option is checked, the computer 101 can determine that the automatic connection option is turned on. If the automatic connection option is not checked, the computer 101 can determine that the automatic connection option is not turned on.

[0239] When the automatic connection option is turned on, computer 101 can automatically connect to device E and realize keyboard and mouse pass-through; if the automatic connection option is not turned on, computer 101 cannot realize keyboard and mouse pass-through with device E.

[0240] S1203: If the automatic connection option is not enabled, a user operation of enabling the automatic connection option or a user operation of clicking to connect to device E on the visual interface is detected.

[0241] The user checks the automatic connection option in the super keyboard and mouse interface, and the computer 101 can detect the user's operation of turning on the automatic connection option and automatically connect to the device E. Alternatively, the user clicks the operation of connecting to the device E in the super keyboard and mouse interface (i.e., the visual interface), and the computer 101 can detect the user's operation of connecting to the device E and connect to the device E.

[0242] S1204 : If the automatic connection option is turned on, determine the position of device E relative to computer 101 .

[0243] The position of device E relative to computer 101 can be on the left, right or middle, which is not limited in this embodiment of the present application.

[0244] The computer 101 can determine the position of the device E relative to the computer 101 through a visual interface or ultrasonic method.

[0245] S1205 : Determine the direction of travel from the computer 101 to the device E based on the position of the device E relative to the computer 101 .

[0246] If device E is on the left side relative to computer 101, computer 101 can determine that the direction of travel from computer 101 to device E is left travel, that is, the mouse travels from the left side of the computer 101 screen. If device E is on the right side relative to computer 101, computer 101 can determine that the direction of travel from computer 101 to device E is right travel, that is, the mouse travels from the right side of the computer 101 screen. If device E is in the middle relative to computer 101, computer 101 can determine that the direction of travel from computer 101 to device E is the default direction. The default direction of travel can be left travel or right travel, which is not limited in this embodiment of the present application.

[0247] S1206: If the mouse crossing operation is consistent with the crossing direction, the keyboard and mouse crossing is realized.

[0248] If the direction of the mouse traversal from computer 101 to device E is left traversal, and the mouse traversal operation is to move the mouse to the left at the left edge of the screen, keyboard and mouse traversal is achieved. If the direction of the mouse traversal from computer 101 to device E is left traversal, and the mouse traversal operation is to move the mouse to the right at the right edge of the screen, keyboard and mouse traversal is not achieved.

[0249] If the direction of the mouse traversal from computer 101 to device E is right traversal, and the mouse traversal operation is to move the mouse to the right at the right edge of the screen, keyboard and mouse traversal is achieved. If the direction of the mouse traversal from computer 101 to device E is right traversal, and the mouse traversal operation is to move the mouse to the left at the left edge of the screen, keyboard and mouse traversal is not achieved.

[0250] S1207 , detecting that the position of device E relative to computer 101 has changed.

[0251] The computer 101 can determine whether the position of the device E relative to the computer 101 has changed through a visual interface or ultrasonic method. The specific implementation method can be as shown in Figures 9 and 10 above, and will not be repeated here.

[0252] S1208. Update the crossing direction according to the changed position of the device E.

[0253] Computer 101 can change the crossing direction according to the above-mentioned rules, which will not be described in detail here.

[0254] The keyboard and mouse traversal control method provided in the embodiment of the present application supports manual and automatic connection of traversable devices. At the same time, the traversal direction can be determined according to the position of the traversable device, so that different devices can share the mouse and keyboard, which is conducive to achieving seamless access between different devices.

[0255] The method provided in the embodiments of the present application can be applied to personal computers, such as desktop computers, laptop computers (such as 101 above), small laptop computers, tablet computers, and ultrabooks. The software system of the personal computer is a Windows system. The device that realizes keyboard and mouse crossover with the personal computer can be a device that deploys the Android system, such as a mobile phone (such as the mobile phone 102 above) or a tablet computer (such as the mobile phone 103 above). Among them, the device that deploys the Android system can be referred to as an Android device. For the convenience of description, it is referred to as an Android device in the following embodiments.

[0256] In order to better understand the embodiments of the present application, the software structure of the PC and Android device in the embodiments of the present application is introduced below.

[0257] For example, a software structure diagram of a PC is shown in Figure 13. As shown in Figure 13, the PC includes a service layer, a capability layer, a driver layer, and a hardware layer.

[0258] The business layer includes the human-computer interaction (UI) interface implementation module, setting management module, state management module, main service module, direction recognition module, connection discovery module, rule management module, keyboard and mouse business module and drag management module.

[0259] The UI module is used to provide a visual interface for users. For example, the UI module can display the main interface shown in Figure 1, the PC Manager interface shown in Figure 5, and the interfaces including bubble boxes shown in Figures 6, 7, and 8. The settings management module is used to manage the PC Manager's settings. For example, the settings management module can provide users with the various options in PC Manager shown in Figure 5. The status management module is used to manage information about traversable devices and keyboard and mouse traversal directions. For example, the status management module can determine the number, location, and traversal directions of traversable devices. The main service module can provide autonomously initiated services for the keyboard and mouse service. The direction recognition module is used to invoke direction recognition capabilities to identify the direction of the Android device relative to the PC, monitor direction changes, and save direction information. The connection discovery module is used to discover and connect to connectable Android devices. The rule management module is used to manage rules for connecting to Android devices and rules for changing traversal directions. The keyboard and mouse service module is used to invoke, trigger, or disable keyboard virtualization capabilities. The drag management module is used to monitor user drag operations.

[0260] The capability layer includes direction recognition capabilities, a keyboard virtualization capability module, a keyboard and mouse event detection module, a device management middleware module, a communication module (Magic Link), a PC manager framework, an account system, a trust ring, a multimodal sensor data platform (MSDP), Bluetooth, an audio system, and a storage system. The keyboard virtualization capability module is used to capture and transmit keyboard and mouse events, and can also be used to adapt the mouse position and movement speed for devices with different resolutions. The keyboard and mouse event detection module is used to capture keyboard and mouse events. The device management middleware module provides data sharing capabilities between devices, such as the ability to enable or disable services between devices. The communication module supports the establishment of connection channels between devices and the transmission of data capabilities. The PC manager framework manages PC manager options. The account system is used to obtain PC login accounts. The trust ring ensures that different devices use the same login account. MSDP detects when a device transitions from motion to inactivity. The audio system manages high-performance audio playback. The storage system stores data.

[0261] The aforementioned direction recognition capabilities include distance and orientation fencing, motion detection, Bluetooth measurement, and ultrasonic measurement. The ultrasonic measurement capability includes an ultrasonic algorithm. The direction recognition capability can be used to measure the direction of an Android device relative to a PC.

[0262] The aforementioned keyboard and mouse virtualization capability module includes a keyboard and mouse virtualization service module, a keyboard and mouse interception module, a keyboard and mouse capture module, an edge computing module, a conversion module, a protocol management module, and a data transmission module. The keyboard and mouse virtualization service module serves as the entry point for the keyboard and mouse virtualization service. It provides access to the keyboard and mouse service module development interface to start or end the service, schedule the relationship between keyboard and mouse event modules, and manage their status. The keyboard and mouse interception module prevents the PC from continuing to respond to keyboard and mouse events after keyboard and mouse traversal is complete. The keyboard and mouse capture module captures keyboard and mouse events. The edge computing module determines whether the mouse is at the edge based on its position. The conversion module matches the coordinates of the PC and Android device. For example, the conversion module converts the mouse traversal position on the PC screen to the position of the mouse on the Android device screen. The conversion module converts the mouse movement speed on the PC screen to the movement speed of the mouse on the Android device screen. The protocol management module packages the intercepted keyboard and mouse events using a preset protocol and sends the packaged keyboard and mouse events to the data transmission module. The data transmission module then transmits the packaged keyboard and mouse events to the Android device.

[0263] The driver layer includes Bluetooth driver, audio driver, Wi-Fi driver and human interface device (HID) driver.

[0264] The hardware layer includes an accelerometer and a gyroscope. The PC can determine the device's motion changes through the acceleration data obtained by the accelerometer and the angular acceleration data obtained by the gyroscope.

[0265] Figure 14 shows a software structure diagram of an Android device. As shown in Figure 14, the Android device includes an application layer, an application framework layer, an Android runtime and system library, and a kernel layer.

[0266] The application layer includes: UI module, settings management module, status management module, main service module, direction recognition module, connection discovery module, keyboard and mouse service module, direction recognition capability, keyboard virtualization capability module, keyboard and mouse event detection module, device management middleware module, communication module (magic link), account system, trust ring, service framework and drag management module.

[0267] The UI module provides a visual interface for users. For example, the UI module displays the main interface and application interfaces. The settings management module configures keyboard and mouse functions. The state management module manages state changes during the traversal process, such as whether a traversal has been completed or not. The main service module provides autonomously initiated services for the keyboard and mouse service. The direction recognition module provides direction recognition capabilities. The discovery connection module manages connection capability scheduling and data communication within the keyboard and mouse service. The rule management module manages rules for connecting to Android devices and changing traversal directions. The keyboard and mouse service module invokes, triggers, or disables keyboard virtualization capabilities. The drag management module invokes the drag framework to monitor user drag operations. The service framework manages keyboard and mouse service capability switch registration. The keyboard and mouse virtualization capability module provides keyboard and mouse virtualization capabilities. The keyboard and mouse event detection module captures keyboard and mouse events. The device management middleware provides data sharing capabilities between devices. The communication module provides connection channels and data transmission between devices.

[0268] It should be noted that the communication module is a module for communication between different devices. Data obtained by other modules must first be transmitted to the communication module before being transmitted to other devices. In subsequent embodiments, to simplify the steps, in addition to communication modules, when other modules communicate across devices, it can be understood that the other modules first transmit the data to the communication module, which then transmits it to the communication module of the other device, and then the communication module of the other device transmits it to the module that receives the data.

[0269] The above-mentioned keyboard and mouse virtualization capability module includes a keyboard and mouse virtualization service module, an edge computing module, a conversion module, a protocol parsing management module, a data transmission module and a virtual device management module. Among them, the keyboard and mouse virtualization service module, the edge computing module and the conversion module have similar functions to those in Figure 13 above, and will not be repeated here. The protocol parsing management module is used to parse the packaged keyboard and mouse events to obtain keyboard and mouse events. The data transmission module is used to receive the packaged keyboard and mouse events and transmit them to the protocol parsing management module. The virtual device management module is an operation interface module for virtual input (Uinput), including creating / closing keyboard and mouse Uinput devices, providing a keyboard and mouse event control interface and performing keyboard and mouse event input control.

[0270] It should be noted that when the keyboard and mouse virtualization capability module transmits data, it needs to first transmit it to the keyboard and mouse business module, and then transmit it to other modules. In order to simplify the steps, the keyboard and mouse virtualization capability module directly transmits data to other modules. It can be understood that the keyboard and mouse virtualization capability module first transmits the data to the keyboard and mouse business module, and then the keyboard and mouse business module transmits the data to other modules.

[0271] The aforementioned direction recognition capabilities include distance and direction fences, motion detection, Bluetooth measurement, and ultrasonic measurement. The ultrasonic measurement capability includes an ultrasonic algorithm. The direction recognition capability can be used to measure the PC's direction relative to the Android device.

[0272] The relationship between the direction recognition capability and the direction recognition module is similar and will not be repeated here.

[0273] This direction recognition capability works in conjunction with the direction recognition capability in FIG. 13 to determine the distance and direction of the Android device relative to the PC.

[0274] For example, when the Android device's MSDP detects that the Android device has transitioned from motion to stillness, it can report this to the Android device's distance and direction fence via the Android device's motion detection meta-capability. The Android device's distance and direction fence can initiate an ultrasonic measurement based on this message, essentially initiating local recording via the Android device's ultrasonic measurement meta-capability. The audio can be high-performance audio. The Android device's ultrasonic measurement meta-capability can instruct the communication module to notify the PC to begin direction measurement. The PC's communication module receives the measurement message and initiates the measurement process. The PC's communication module can record audio using its ultrasonic measurement meta-capability and return a confirmation message to the Android device's communication module. The PC plays the audio, and the Android device also begins playing audio based on the confirmation message. The PC's communication module sends the measurement results to the Android device's communication module. The Android device receives the PC's measurement results and combines them with local measurement results to determine the final distance and direction. The Android device and PC's ultrasonic measurement meta-capabilities can report the measurement results to the distance and direction fence. The Android device and / or PC's distance and direction fence alerts the user when the device detects a device within a preset distance.

[0275] The application framework layer includes a drag framework, a multi-screen framework, an input management module, and a storage system. The drag framework provides dragging capabilities for super keyboard and mouse traversal. The input management module provides keyboard and mouse event reporting. The input management module may include a keyboard and mouse framework and an input framework. The keyboard and mouse framework reports keyboard and mouse events, while the input framework reports input events. The storage system stores data. The multi-screen framework provides keyboard and mouse multi-screen operation.

[0276] The Android runtime and system libraries include the core library, virtual machine, surface manager, media library, and 3D graphics library. The Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one for the Java language's callable functions and the other for the Android core library. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files in the application layer and application framework layer as binary files. The virtual machine is responsible for performing functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection. The system library can include modules for multiple functions, such as the surface manager, media library, and 3D graphics library.

[0277] The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as static image files. It also supports various audio and video encoding formats, such as JPG and PNG. The 3D graphics processing library implements 3D graphics drawing, image rendering, compositing, and layer processing.

[0278] The kernel layer is the layer between hardware and software. It drives the hardware, making it work. It includes the Bluetooth driver, audio driver, Wi-Fi driver, virtual input driver (Uinput driver), and HID driver. The virtual input driver is used to inject input events.

[0279] In order to better understand the embodiments of the present application, the embodiments of the present application provide a detailed introduction to the methods provided by the embodiments of the present application in combination with the modules involved in the above-mentioned Figures 13 and 14.

[0280] In order to realize the keyboard and mouse pass-through function, after the PC is turned on, it can store the device information of the online Android devices through table tA, store the device information of the online Android devices with the keyboard and mouse function turned on through table tB, and store the device information of the PC connected through table tC to better support the keyboard and mouse pass-through function.

[0281] For example, Figure 15 shows a schematic flow chart of a startup method 1500 in a keyboard and mouse scenario. This method 1500 can be applied to a communication system including a PC and an Android device, such as the aforementioned communication system 100, but the embodiments of the present application are not limited thereto. The software architecture diagram of the PC can be shown in Figure 13, and the software architecture diagram of the Android device can be shown in Figure 14, but the embodiments of the present application are not limited thereto.

[0282] The method 1500 may include the following steps:

[0283] S1501: The keyboard and mouse service module of the PC is started.

[0284] The keyboard and mouse service module of the PC can be automatically started when the PC is turned on.

[0285] S1502. The keyboard and mouse service module of the PC sends instruction information A to the keyboard and mouse virtualization capability module of the PC. The instruction information A is used to instruct the keyboard and mouse virtualization capability module of the PC to start the service. Correspondingly, the keyboard and mouse virtualization capability module of the PC receives the instruction information A.

[0286] The instruction information A is used to instruct the keyboard and mouse virtualization capability module of the PC to start the service, that is, to enable or function the keyboard and mouse virtualization capability module of the PC.

[0287] The keyboard and mouse service module of the PC turns on the keyboard and mouse virtualization capability module of the PC through the instruction information A. It can also be understood that the keyboard and mouse service module of the PC calls the keyboard and mouse virtualization capability module of the PC and obtains the information returned by the keyboard and mouse virtualization capability module of the PC.

[0288] S1503. The keyboard and mouse virtualization capability module of the PC records the keyboard and mouse crossing status based on the indication information A, and monitors whether the keyboard and mouse have crossed.

[0289] The pass-through status can be either Yes or No. If the pass-through status is Yes, it means the mouse pointer has passed from the PC to the Android device. If the pass-through status is No, it means the mouse pointer has not passed from the PC to the Android device.

[0290] A criterion for determining keyboard and mouse rollover can be that the mouse pointer touches the side edge of the PC screen twice in a row. Based on this criterion, the PC's keyboard and mouse virtualization module can monitor whether keyboard and mouse rollover has occurred. If the mouse pointer touches the side edge of the PC screen twice in a row, the PC's keyboard and mouse virtualization module can determine that keyboard and mouse rollover has occurred.

[0291] The PC's keyboard and mouse virtualization capability module can also transmit keyboard and mouse cross-border information to the keyboard and mouse service module.

[0292] S1504: The keyboard and mouse service module of the PC sends a request message a to the communication module of the PC, wherein the request message a is used to request the communication module of the PC to notify the detected online Android device. Correspondingly, the communication module of the PC can receive the request message a.

[0293] When an Android device goes online, it can be understood that the Android device and PC are logged in using the same account.

[0294] The communication module of the PC can detect whether there is an online Android device. If there is an online Android device, based on the request message a, the device identity (ID) of the online Android device is sent to the keyboard and mouse service module of the PC.

[0295] The keyboard and mouse service module of the PC sends a request message a to the communication module of the PC. It can also be understood that the keyboard and mouse service module of the PC registers the device offline fence in the communication module of the PC. When the communication module of the PC detects that there is an online device, it returns the device ID to the keyboard and mouse service module of the PC.

[0296] It should be noted that the above S1502 and S1504 are steps that will be executed after the keyboard and mouse service module of the PC is started. The embodiment of the present application does not limit the order in which these two steps are executed.

[0297] S1505. The keyboard and mouse service module of the PC sends a request message b to the direction identification module of the PC, requesting the direction identification module of the PC to provide the detected direction information of the Android device relative to the PC. In response, the direction identification module of the PC receives the request message b. The direction information of the Android device relative to the PC may include at least one of a coordinate angle, a coordinate direction, a distance, or a traversing direction of the Android device relative to the PC.

[0298] The PC's keyboard and mouse service module sends a request message b to the PC's direction identification module. This can also be understood as the PC's keyboard and mouse service module instructing the PC's direction identification module to register a direction fence. When the PC's direction identification module detects the direction of the Android device relative to the PC, it can send direction information to the PC's keyboard and mouse service module.

[0299] Optionally, the direction identification module of the PC can detect the Android device in the trust ring, and can also send the device ID of the Android device to the keyboard and mouse service module of the PC. If the device ID received by the keyboard and mouse service module of the PC from the direction identification module of the PC is different from the device ID of the communication module of the PC, the device ID returned by the communication module of the PC shall prevail.

[0300] With this implementation, both the PC's direction recognition module and the PC's communication module can return the device ID within the trust ring. If the returned device IDs are the same, the devices within the trust ring can be determined more accurately, which helps the PC connect to the Android device more accurately.

[0301] Optionally, the PC's direction recognition module can detect Android devices within a preset distance. That is, if an Android device is within the PC's near field, the PC's direction recognition module can send the Android device ID to the PC's keyboard and mouse service module. The preset distance can be 50 centimeters, but embodiments of the present application are not limited thereto.

[0302] With this implementation, the PC can connect to Android devices within the near field range and the trust ring, eliminating keyboard and mouse crossing with distant devices, which is more in line with the user experience.

[0303] Optionally, the PC's direction recognition module can use the communication module to determine whether the Android device's screen is in the on-screen state. If so, the PC sends the Android device's ID to the PC's keyboard and mouse service module. In other words, the PC can obtain the Android device's device ID when the screen is in the on-screen state. Otherwise, the subsequent steps are not performed.

[0304] In this implementation, when the screen is on, it indicates that the PC is in use. The device ID of the Android device is obtained to facilitate subsequent device connection, which is more in line with user usage habits. When the screen is black, the device ID of the Android device is not obtained, which helps save PC power consumption.

[0305] Optionally, the direction recognition module of the PC can obtain the device ID of an Android device whose screen is in a bright state, is in a trust circle, and is within a preset distance.

[0306] S1506: The service framework of the Android device is started.

[0307] The service framework of Android devices can start automatically when the device is booted.

[0308] The service framework of an Android device can be used to monitor the on / off status and on / off change information of the keyboard and mouse functions of the Android device. The keyboard and mouse function refers to the keyboard and mouse pass-through function, and the on / off status of the keyboard and mouse function refers to whether the keyboard and mouse function is on or off.

[0309] For example, when the keyboard and mouse function is turned off, the service framework of the Android device can determine that the keyboard and mouse function state is off. If it is detected that the user turns on the keyboard and mouse function, the service framework of the Android device can determine that the keyboard and mouse function state is on.

[0310] S1507. The service framework of the Android device sends the keyboard and mouse function status to the device management middleware module of the Android device. Correspondingly, the device management middleware module of the Android device receives the keyboard and mouse function status.

[0311] For example, if the service framework of the Android device determines that the keyboard and mouse function is in the off state, the service framework sends information indicating that the keyboard and mouse function is in the off state to the device management middleware module of the Android device. If the service framework of the Android device determines that the keyboard and mouse function is in the on state, the service framework sends information indicating that the keyboard and mouse function is in the on state to the device management middleware module of the Android device. The keyboard and mouse function is turned on or off by the user. If the user turns on the keyboard and mouse function, the service framework of the Android device can determine that the keyboard and mouse function is in the on state in response to the user's operation of turning on the keyboard and mouse function; if the user turns off the keyboard and mouse function, the service framework of the Android device can determine that the keyboard and mouse function is in the off state in response to the user's operation of turning off the keyboard and mouse function.

[0312] S1508: If the service framework of the Android device detects that the keyboard and mouse function status has changed, the service framework of the Android device sends the keyboard and mouse function change status to the device management middleware module of the Android device. Correspondingly, the device management middleware module of the Android device receives the keyboard and mouse function change status and updates the keyboard and mouse function status.

[0313] For example, if the service framework of the Android device detects that the state of the keyboard and mouse function changes from an off state to an on state, the service framework of the Android device sends information that the state of the keyboard and mouse function is on to the device management middleware module of the Android device, and the device management middleware module of the Android device can update the state of the keyboard and mouse function to an on state. If the service framework of the Android device detects that the state of the keyboard and mouse function changes from an on state to an off state, the service framework of the Android device sends information that the state of the keyboard and mouse function is off to the device management middleware module of the Android device, and the device management middleware module of the Android device can update the state of the keyboard and mouse function to an off state.

[0314] It should be noted that there is no order restriction between the above S1506 to S1508 and S1501 to S1505.

[0315] S1509 : The communication module of the PC and the communication module of the Android device discover each other, forming a trust loop.

[0316] The PC and the Android device use the same account to log in. At this time, the communication module of the PC and the communication module of the Android device can sense and discover each other, forming a trust loop. The communication module of the PC can obtain the device ID of the Android device.

[0317] S1510: A trust ring is formed. The communication module of the PC detects that the Android device is online and can send the device ID of the Android device to the keyboard and mouse service module of the PC. Correspondingly, the keyboard and mouse service module of the PC receives the device ID.

[0318] The device ID can be at least one of numbers, letters, or symbols, which is not limited in this embodiment of the present application.

[0319] There may be one or more online Android devices, which is not limited in this embodiment of the present application. If there are multiple online Android devices, the communication module of the PC may send the device IDs of the multiple Android devices to the keyboard and mouse service module of the PC.

[0320] S1511. The keyboard and mouse service module of the PC adds the device ID to table tA based on the device ID.

[0321] Table tA is used to store device information of online Android devices, wherein the device information is not limited to the device ID.

[0322] The keyboard and mouse service module of the PC can create table tA and add the device ID in table tA. In the subsequent process, if there is a new online device, table tA can be updated, that is, the device ID of the online device can be added to table tA.

[0323] Optionally, the information in table tA can be added by the above-mentioned status management module. The keyboard and mouse service module of the PC can send an instruction information for adding the device ID to the status management module based on the device ID. The status management module adds the device ID to table tA based on the instruction information.

[0324] In one embodiment, table tA may include a device identifier and the connection status between the Android device and the PC. The connection status between the Android device and the PC can be updated in real time. When the PC and the Android device are not connected or disconnected, the connection status in table tA may be "not connected." When the PC and the Android device are connected or disconnected and then reconnected, the connection status in table tA may be "connected."

[0325] For example, table tA may be as shown in Table 2.

[0326] Table 2

[0327] As shown in Table 2, the device ID of the online Android device is 11_A, and the connection status between the PC and the Android device is disconnected.

[0328] S1512: The keyboard and mouse service module of the PC sends instruction information B to the device management middleware module of the PC. Instruction information B is used to instruct the device management middleware module to determine whether the keyboard and mouse function of the Android device is enabled based on the device ID. Correspondingly, the device management middleware module of the PC receives instruction information B. Instruction information B may include the device ID.

[0329] S1513 . The device management middleware module of the PC synchronizes information with the device management middleware module of the Android device based on the instruction information B.

[0330] The information synchronized between the device management middleware module of the PC and the device management middleware module of the Android device includes the keyboard and mouse function status.

[0331] The device management middleware module of the Android device can send the updated keyboard and mouse function status to the device management middleware module of the PC. Correspondingly, the device management middleware module of the PC can also send the updated keyboard and mouse function status to the device management middleware module of the Android device. The method by which the device management middleware module of the PC obtains the keyboard and mouse function status is the same as that of the device management middleware module of the Android device, and will not be repeated here.

[0332] If the number of device IDs in the indication information B is multiple, the device management middleware module of the PC can synchronize information with the device management middleware modules of multiple Android devices based on the indication information B.

[0333] S1514. Based on the synchronized information, the device management middleware module of the PC can determine whether the keyboard and mouse functions of the Android device are enabled. If the keyboard and mouse functions are enabled on the Android device, the device management middleware module of the PC can send a keyboard and mouse function enablement message to the keyboard and mouse service module of the PC. In response, the keyboard and mouse service module of the PC receives this message and can determine that the keyboard and mouse functions of the Android device are enabled.

[0334] If the device management middleware module of the PC can determine that the keyboard and mouse function of the Android device is not enabled based on the synchronized information, the device management middleware module of the PC may not send the keyboard and mouse information to the keyboard and mouse service module of the PC and not execute subsequent steps.

[0335] S1515. The keyboard and mouse service module of the PC adds the device ID to table tB based on the keyboard and mouse function activation information.

[0336] Table tB may include device information of Android devices with keyboard and mouse functions enabled and within the trust circle. The device information in Table tB is not limited to the device ID.

[0337] Optionally, the information in table tB can be added by the above-mentioned status management module. The keyboard and mouse service module of the PC can send an indication information of adding the device ID to the status management module based on the keyboard and mouse function activation information. The status management module adds the device ID in table tB based on the indication information.

[0338] S1516. The direction recognition module of the PC obtains angle information through the ultrasonic sensor, and determines the direction information of the Android device relative to the PC based on the angle information.

[0339] The ultrasonic sensor can be used to detect the angle of the Android device relative to the ultrasonic sensor. The PC's direction recognition module can determine the direction of the Android device relative to the PC based on this angle information. The PC's direction recognition module can determine the coordinate direction and distance of the Android device relative to the PC based on the coordinate angle. For details, see Figure 9 above and will not be further described here. The PC's direction recognition module can also determine a recommended crossing direction for the Android device based on the coordinate direction.

[0340] The recommended crossing direction can be determined based on the coordinate direction. For example, if the coordinate direction is left, the recommended crossing direction is left crossing; if the coordinate direction is right, the recommended crossing direction is right crossing; if the coordinate direction is middle, the recommended crossing direction is the default crossing direction. The default crossing direction can be left crossing or right crossing.

[0341] Optionally, if the coordinate direction is middle, the suggested crossing direction can also be related to the previous crossing direction. If the coordinate direction of the Android device relative to the PC is middle, the suggested crossing direction can be the previous crossing direction. For example, if the previous crossing direction was left, the suggested crossing direction is left. If the previous crossing direction was right, the suggested crossing direction is right.

[0342] S1517: The direction identification module of the PC sends direction information to the keyboard and mouse service module of the PC. Correspondingly, the keyboard and mouse service module of the PC receives the direction information.

[0343] S1518. The keyboard and mouse service module of the PC may add direction information to table tB.

[0344] The PC's keyboard and mouse service module can instruct the status management module to add direction information to table tB. Table tB can include the device ID and direction information relative to the PC for Android devices with keyboard and mouse functions enabled and within the trusted ring. Table tB can also include connection status.

[0345] For example, table tB may include information such as the device ID, direction information, and connection status of an Android device with keyboard and mouse functions enabled and within the trusted circle relative to the PC. The direction information includes coordinate direction, coordinate angle, distance, and traversal direction. Table tB may be shown in Table 3.

[0346] Table 3

[0347] As shown in Table 3, the device ID of the Android device with the keyboard and mouse functions enabled and within the trust circle is 11_A, the coordinate direction relative to the PC is left, the coordinate angle is 60°, the distance relative to the PC is 40 cm, the crossing direction is left crossing, and the connection status with the PC is disconnected.

[0348] Optionally, table tB may also include the screen on / off status of the Android device, so as to implement keyboard and mouse cross-talk when the screen is on, and not implement keyboard and mouse cross-talk when the screen is off to save power consumption.

[0349] S1519: The keyboard and mouse service module of the PC may send an instruction message C to the direction recognition module of the PC, wherein the instruction message C is used to instruct the direction recognition module of the PC to monitor the coordinate direction change of the Android device. Correspondingly, the direction recognition module of the PC receives the instruction message tB.

[0350] S1520. The direction recognition module of the PC monitors the change in the coordinate direction of the Android device based on the indication information C.

[0351] The direction recognition module of the PC can monitor the coordinate direction of the Android device in real time based on the indication information C.

[0352] S1521. The direction recognition module of the PC may send the coordinate direction change information to the keyboard and mouse service module of the PC. Correspondingly, the keyboard and mouse service module of the PC receives the coordinate direction change information.

[0353] The coordinate direction change information is the coordinate direction information after the change. For example, if the coordinate direction changes to the right, the coordinate direction change information is right.

[0354] If the crossing direction or distance of the Android device changes, the direction recognition module of the PC may also send the crossing direction change information or the changed distance to the keyboard and mouse service module of the PC.

[0355] S1522. The keyboard and mouse service module of the PC may update the coordinate direction information in table tB based on the coordinate direction change information.

[0356] The keyboard and mouse service module of the PC may also update the crossing direction or distance in table tB when the crossing direction or distance changes.

[0357] Optionally, the keyboard and mouse service module of the PC may also update the connection status and screen on / off status in table tB.

[0358] S1523. The direction identification module of the PC may further store device information of the connected device in table tC.

[0359] The direction recognition module of the PC detects that there is an Android device connected to the PC and can store the device information of the connected device in table tC.

[0360] Table tC is used to store information about devices connected to the PC. Table tC may include information such as device ID and traversal direction.

[0361] Optionally, table tC may also include at least one of the following information: the coordinate direction, coordinate angle, whether it is the first crossing, or the current crossing status of the connected device. The information on whether it is the first crossing and the current crossing status in table tC are determined by the keyboard and mouse virtualization capability module of the PC. The keyboard and mouse virtualization capability module of the PC monitors the mouse crossing, determines the device ID of the crossing device, records the number of crossings to the device, and updates the crossing status. The keyboard and mouse virtualization capability module of the PC may send the device ID, the number of crossings to the device, and the updated crossing status to the direction identification module of the PC, which saves this information to table tC.

[0362] For example, table tC may include device ID, crossing direction, coordinate direction of connected devices, coordinate angle, whether it is the first crossing, and current crossing status. Table tC may be as shown in Table 4.

[0363] Table 4

[0364] As shown in Table 4, the device ID of the Android device connected to the PC is 11_A, the coordinate direction relative to the PC is left, the crossing direction from the PC to the Android device is left crossing, the coordinate angle of the Android device relative to the PC is 60°, the crossing from the PC to the Android device using the keyboard and mouse is the first crossing, and the current crossing status is crossing in progress.

[0365] In the startup method provided in the embodiment of the present application, the PC can use table tA to save the device information of the online device, table tB to save the device information of the connectable (or traversable) device, and table tC to save the device information of the connected device, so as to facilitate subsequent query of the information in the table. The Android device can monitor the on and off status of the keyboard and mouse functions to facilitate subsequent keyboard and mouse traversal.

[0366] After completing method 1500, the PC can connect to the Android device to enable keyboard and mouse passthrough. If the first option in Figure 5, "Tap the mouse pointer twice on the side edge of the screen to establish a connection," is selected, the PC can automatically connect to the Android device. If the first option in Figure 5 is not selected, the PC can manually connect to the Android device.

[0367] In one example, the first option in FIG5 above, “touch the mouse pointer twice on the side edge of the screen to establish a connection,” is selected, and the PC can automatically connect to the Android device.

[0368] The trigger condition for the PC to automatically connect to the Android device is that the user touches the mouse pointer on the side edge of the screen twice in a row. After the above method 1500, the PC detects that the user touches the mouse pointer on the side edge of the screen twice in a row and can execute the method for automatically connecting to the Android device.

[0369] For example, Figure 16 shows a schematic flow chart of an automatic connection method 1600. This method 1600 can be applied to a communication system including a PC and an Android device, such as the aforementioned communication system 100, but embodiments of the present application are not limited thereto. The software architecture diagram of the PC can be shown in Figure 13, and the software architecture diagram of the Android device can be shown in Figure 14, but embodiments of the present application are not limited thereto.

[0370] The method 1600 may include the following steps:

[0371] S1601: After S1523, the keyboard and mouse virtualization module of the PC detects that the automatic connection option is enabled. When the automatic connection option is enabled, the keyboard and mouse virtualization module of the PC detects that the user touches the mouse pointer twice consecutively on the side edge of the screen, i.e., monitors keyboard and mouse crossover.

[0372] The automatic connection option is similar to the "touch the mouse pointer twice on the side edge of the computer screen to establish a connection" option in Figure 5 above, but the present application is not limited thereto. If the automatic connection option is turned on, it indicates that the PC supports automatic connection to Android devices, and you can continue with the subsequent steps. If the automatic connection option is turned off, it indicates that the PC does not support automatic connection to Android devices, and you can stop the subsequent steps.

[0373] For example, the coordinate of the side edge is 0, and the keyboard and mouse virtualization capability module of the PC detects that the coordinate of the mouse pointer is 0 twice within 3 seconds, then it can be determined that the keyboard and mouse have crossed.

[0374] S1602: The keyboard and mouse virtualization capability module of the PC sends keyboard and mouse cross-talk information to the keyboard and mouse service module of the PC. Correspondingly, the keyboard and mouse service module of the PC receives the keyboard and mouse cross-talk information.

[0375] The information of keyboard and mouse pass-through is used to indicate keyboard and mouse pass-through. In one embodiment, the information of keyboard and mouse pass-through can be a flag. For example, the information of keyboard and mouse pass-through is represented by a binary character 1 to indicate keyboard and mouse pass-through.

[0376] S1603: The keyboard and mouse service module of the PC generates a sidebar at the side edge of the keyboard and mouse based on the keyboard and mouse traversal information.

[0377] The sidebar can be the bubble box mentioned above.

[0378] S1604: The keyboard and mouse service module of the PC determines whether the smart capability is enabled.

[0379] The smart capability options include the “automatically adjust screen arrangement as the device placement changes” option in FIG5 above, but the present application is not limited thereto.

[0380] Smart capabilities refer to whether the PC can automatically detect the Android device's orientation relative to the PC. If Smart capabilities are enabled, the PC can automatically detect the Android device's orientation relative to the PC and update the aforementioned table tB in real time. If Smart capabilities are disabled, the PC cannot automatically detect the Android device's position and distance relative to the PC, and Table tB cannot be updated in real time. Table tB cannot be used in subsequent processes to reduce errors.

[0381] It should be noted that after the PC's keyboard and mouse service module is started, it can register for smart capability monitoring to obtain the status of the smart capability on or off, and can also register for smart capability switch change monitoring to sense changes in the smart capability switch status.

[0382] If the PC's keyboard and mouse service module detects that the smart capability has changed from the off state to the on state, the PC's keyboard and mouse service module can re-register the smart capability switch change monitor to monitor the smart capability switch state. If the PC's keyboard and mouse service module detects that the smart capability has changed from the on state to the off state, the information in table tB is cleared.

[0383] S1605: If the smart capability is enabled, the keyboard and mouse service module of the PC determines the connectable device through table tB.

[0384] The keyboard and mouse service module of the PC can determine the connectable devices in table tB based on the traversal direction of the keyboard and mouse.

[0385] For example, table tB includes a crossing direction, and the keyboard and mouse service module of the PC may determine a device whose crossing direction in table tB is consistent with the crossing direction of the keyboard and mouse as a connectable device.

[0386] The connectable device may be one or more, and this embodiment of the present application does not limit this. When there is only one connectable device, the PC can directly connect to the connectable device. When there are multiple connectable devices, the PC can display the icons and / or models of the connectable devices in the above-mentioned sidebar so that the user can select a device for connection.

[0387] S1606. The keyboard and mouse service module of the PC sends a request message c to the communication module of the PC. The request message c is used to request the communication module of the PC to establish a connection with the communication module of the Android device.

[0388] The Android device is a connectable device determined by the keyboard and mouse service module of the PC. After the communication module of the PC establishes a connection with the communication module of the Android device, subsequent keyboard and mouse crossing is facilitated.

[0389] S1607 : The communication module of the PC establishes a message channel session and a file transfer channel session with the communication module of the Android device based on the request message c.

[0390] The communication module of the PC establishes a message channel session and a file transfer channel session with the communication module of the Android device, so as to facilitate the transmission of messages and files between the two devices through keyboard and mouse operations.

[0391] S1608: If the communication module of the PC successfully establishes a channel session with the communication module of the Android device, the communication module of the PC sends a session success message to the keyboard and mouse service module of the PC.

[0392] S1609. The communication module of the Android device starts the service for the keyboard and mouse service module of the Android device.

[0393] The communication module of the Android device pulls up the service of the keyboard and mouse business module, that is, makes the keyboard and mouse business module work to support keyboard and mouse crossing.

[0394] S1610. The keyboard and mouse service module of the Android device may send a connection success message to the direction recognition module of the PC. Correspondingly, the keyboard and mouse service module of the Android device receives the connection success message.

[0395] The connection success message indicates that the PC and Android device have been successfully connected, and the preparations before the time travel are complete, and the time travel can be carried out.

[0396] S1611. The keyboard and mouse service module of the Android device updates the above table tC based on the connection message.

[0397] Table tC is used to store information about devices connected to the PC. At this time, the Android device is connected to the PC, and the keyboard and mouse service module of the Android device can add the information of the Android device to the above table tC.

[0398] It is understandable that after the keyboard and mouse are transferred from the PC to the Android device, files or messages can be transferred between the PC and the Android device.

[0399] The method provided in the embodiment of the present application can automatically connect to a connectable device after monitoring keyboard and mouse crossing, reducing the user's manual connection operation, simplifying the operation, and helping to improve the user experience.

[0400] In another example, when the first option in FIG. 5 is not selected, the PC can be connected to the Android device manually.

[0401] The PC can provide a visual interface, which can be as shown in Figure 5 above. The interface includes the connectable device Magi4, mobile phone Magi4xy and mobile phone Magi4y. Users can connect the PC to the Android device by clicking or dragging.

[0402] For example, Figure 17 shows a schematic flow chart of a manual connection method 1700. This method 1700 can be applied to a communication system including a PC and an Android device, such as the aforementioned communication system 100, but embodiments of the present application are not limited thereto. The software architecture diagram of the PC can be shown in Figure 13, and the software architecture diagram of the Android device can be shown in Figure 14, but embodiments of the present application are not limited thereto.

[0403] The method 1700 may include the following steps:

[0404] S1701. After S1523, the keyboard and mouse service module of the PC detects the user's operation of opening the visual interface.

[0405] The visual interface allows users to view the connection status of the PC and other devices, select devices to connect, and adjust the position of connected devices.

[0406] S1702 : The keyboard and mouse service module of the PC displays connected devices and online devices on the visual interface in response to the user opening the visual interface.

[0407] The keyboard and mouse service module of the PC can display the connected devices on the visual interface through the above table tC, and can display the online devices on the visual interface through the above table tA.

[0408] For example, the visual interface can be as shown in Figure 5 above, which is also the super keyboard and mouse interface. In the interface shown in Figure 5 above, when the user opens the super keyboard and mouse interface, the keyboard and mouse service module of the PC can detect the user's operation of opening the super keyboard and mouse interface. In response to this operation, the connected devices and online devices are displayed in the super keyboard and mouse interface. As shown in the super keyboard and mouse interface, a tablet is connected to the left side of the PC, and there are three online devices. These three online devices include the mobile phone Magi4, the mobile phone Magi4xy, and the mobile phone Magi4y. The user can select a device from these three online devices to connect, and can also adjust the position of the connected device tablet.

[0409] S1703: The keyboard and mouse service module of the PC detects that the user drags the online device.

[0410] The user can drag the online device to the left, right, or center of the PC on the visual interface. The PC's keyboard and mouse service module can detect the user's operation of dragging the online device.

[0411] For example, in the example shown in FIG5 , the user can drag any one of the three online devices, such as the mobile phone Magi4.

[0412] S1704: The keyboard and mouse service module of the PC responds to the user's operation of dragging the online device to determine whether the smart capability is enabled.

[0413] This step is the same as the above-mentioned S1605 and will not be repeated here.

[0414] S1705: If the smart capability is enabled, the keyboard and mouse service module of the PC determines whether the dragged device meets the connection conditions through the above table tB.

[0415] Table tB includes information such as the coordinate direction, coordinate angle, and connection status of devices that are online and have the keyboard and mouse functions turned on. If the keyboard and mouse service module of the PC detects that there is no information about the device being dragged by the user in Table tB, it can be determined that the keyboard and mouse functions of the device being dragged by the user are not turned on, and the user can be prompted that the keyboard and mouse functions of the device being dragged are not turned on. If the keyboard and mouse service module of the PC detects that there is already a connected device at the location where the device is being dragged, or detects that two devices are already connected, a prompt can be given that the connection cannot be made. It is understandable that the connection conditions may include the following three conditions: 1) the keyboard and mouse functions are turned on; 2) there is no connected device at the dragging location; 3) the number of devices connected to the PC is not greater than 2.

[0416] S1706: If the dragged device meets the connection conditions, the keyboard and mouse service module of the PC sends a connection establishment request to the communication module of the PC. Correspondingly, the communication module of the PC receives the connection establishment request.

[0417] The connection request is also called a request message b, which is used to request the communication module of the PC to establish a connection with the communication module of the dragged device. After the communication module of the PC establishes a connection with the communication module of the Android device, subsequent keyboard and mouse traversal is facilitated.

[0418] S1707: The keyboard and mouse service module of the PC sends information indicating recording of the pass-through status to the keyboard and mouse virtualization capability module of the PC. Correspondingly, the keyboard and mouse virtualization capability module of the PC receives the information indicating recording of the pass-through status.

[0419] The information indicating recording the crossing state is used to instruct the keyboard and mouse virtualization capability module of the PC to record the crossing state. It is understandable that at this time, crossing has not yet occurred, and the keyboard and mouse virtualization capability module of the PC records the crossing state as no or not crossed.

[0420] After 1707, the PC executes the above-mentioned S1608 and S1612, which will not be repeated here.

[0421] The manual connection method provided in the embodiment of the present application can determine the connected device by detecting the user's operation of dragging the online device and connect to it.

[0422] After the PC is successfully connected to the Android device through the manual connection method 1700 or the automatic connection method 1600, the PC and the Android device determine whether the keyboard and mouse are successfully connected. If the connection is successful, the keyboard and mouse are traversed.

[0423] For example, Figure 18 shows a schematic flow chart of a method 1800 for connecting a keyboard and mouse. As shown in Figure 18, this method 1800 can be applied to a communication system including a PC and an Android device, such as the aforementioned communication system 100, but embodiments of the present application are not limited thereto. The software architecture diagram for the PC can be shown in Figure 13, and the software architecture diagram for the Android device can be shown in Figure 14, but embodiments of the present application are not limited thereto.

[0424] The method 1800 may include the following steps:

[0425] S1801. The keyboard and mouse service module of the PC establishes a transmission connection with the keyboard and mouse service module of the Android device.

[0426] After the PC is connected to the Android device, the keyboard and mouse service module of the PC can establish a transmission connection with the keyboard and mouse service module of the Android device. For example, the keyboard and mouse service module of the PC can establish a P2P connection with the keyboard and mouse service module of the Android device.

[0427] S1802. The keyboard and mouse service module of the PC sends instruction information for starting the keyboard and mouse service to the keyboard and mouse service module of the Android device. Correspondingly, the keyboard and mouse service module of the Android device receives the instruction information for starting the keyboard and mouse service.

[0428] The instruction information for starting the keyboard and mouse service is used to instruct the keyboard and mouse service module of the Android device to start the keyboard and mouse service.

[0429] The keyboard and mouse service is also the keyboard and mouse pass-through option. The instruction information for turning on the keyboard and mouse service is to instruct the keyboard and mouse service module of the Android device to turn on the keyboard and mouse pass-through option.

[0430] S1803. The keyboard and mouse service module of the Android device determines whether the keyboard and mouse service is supported based on the instruction information for starting the keyboard and mouse service.

[0431] The keyboard and mouse service is also known as the keyboard and mouse pass-through option. The user can enable or disable the keyboard and mouse service. The Android device's keyboard and mouse service module determines whether the keyboard and mouse service is supported. Specifically, the Android device's keyboard and mouse service module determines whether the user has checked, selected, or enabled the keyboard and mouse pass-through option.

[0432] If the user has checked, selected, or enabled the keyboard and mouse pass-through option, the keyboard and mouse service module of the Android device can support keyboard and mouse services. If the user has not checked, unselected, or disabled the keyboard and mouse pass-through option, the keyboard and mouse service module of the Android device does not support keyboard and mouse services and keyboard and mouse pass-through is not possible.

[0433] S1804. When the keyboard and mouse service is supported, the keyboard and mouse service module of the Android device sends an indication message for starting the keyboard and mouse service to the keyboard and mouse virtualization capability module of the Android device. Correspondingly, the keyboard and mouse virtualization capability module of the Android device receives the indication message for starting the keyboard and mouse service.

[0434] S1805. The keyboard and mouse virtualization capability module of the Android device registers a virtual input (Uinput) based on the instruction information of enabling the keyboard and mouse service.

[0435] Uinput is used to inject keyboard and mouse events.

[0436] S1806. The keyboard and mouse virtualization capability module of the Android device sends a registration result R1 to the keyboard and mouse service module of the Android device. Correspondingly, the keyboard and mouse service module of the Android device receives the registration result R1.

[0437] The registration result R1 can be used to indicate whether the registration Uinput is successful. The registration result can be success or failure. In one embodiment, success and failure can be represented by binary identifiers. For example, 1 indicates success and 0 indicates failure.

[0438] S1807. Based on the indication information of enabling the keyboard and mouse service, the keyboard and mouse virtualization capability module of the Android device sends indication information of registering the cursor position monitor to the multi-screen framework module of the Android device. Correspondingly, the multi-screen framework module of the Android device receives the indication information of registering the cursor position monitor and registers the cursor position monitor.

[0439] After the multi-screen framework module registers the cursor position monitor, it can monitor the cursor position.

[0440] It should be noted that the keyboard and mouse virtualization capability module of the Android device is based on the indication information of the keyboard and mouse service being turned on, and there is no limit on the order between registering Uinput and sending the indication information of registering cursor position monitoring. In other words, the embodiment of the present application does not limit the execution order of the above S1805 and S1807.

[0441] S1808. The keyboard and mouse virtualization capability module of the Android device sends the registration result R2 to the keyboard and mouse service module of the Android device. Correspondingly, the keyboard and mouse service module of the Android device receives the registration result R2.

[0442] The registration result R2 can be used to indicate whether the registration cursor position monitoring is successful. The registration result can be success or failure. In one embodiment, success and failure can be represented by binary identifiers. For example, 1 indicates success and 0 indicates failure.

[0443] S1809. When the keyboard and mouse service is supported, the keyboard and mouse service module of the Android device sends an instruction message for registering a drag monitor to the drag management module of the Android device. Correspondingly, the keyboard and mouse virtualization capability module of the Android device receives the instruction message for registering a drag monitor.

[0444] It should be noted that when keyboard and mouse services are supported, there is no restriction on the order in which the keyboard and mouse service module of the Android device sends the indication information for starting the keyboard and mouse service and sends the indication information for registering the drag and drop monitor. In other words, the embodiment of the present application does not limit the execution order of the above S1804 and S1809.

[0445] S1810. The drag management module of the Android device sends instruction information for registering a drag monitor to the multi-screen framework module of the Android device. Correspondingly, the multi-screen framework module of the Android device receives the instruction information for registering a drag monitor.

[0446] S1811, the keyboard and mouse service module of the Android device saves the processing result R 11 , where the processing result R 11 Including the results of whether the keyboard and mouse service is enabled, whether the Uinput registration is successful, and whether the cursor position monitoring registration is successful.

[0447] For example, the processing result R 11 This may include enabling keyboard and mouse services, successfully registering Uinput, and successfully registering cursor position monitoring.

[0448] S1812, the keyboard and mouse service module of the Android device sends the processing result R to the keyboard and mouse service module of the PC 11, correspondingly, the keyboard and mouse service module of the PC receives the processing result R 11 .

[0449] S1813: The keyboard and mouse service module of the PC sends instruction information for starting the keyboard and mouse service to the keyboard and mouse virtualization capability module of the PC. Correspondingly, the keyboard and mouse service module of the PC receives the instruction information for starting the keyboard and mouse service.

[0450] The above S1802 and S1813 are sent in parallel by the keyboard and mouse service module of the PC, so the above S1802 to S1812 and S1813 to S1818 are executed in parallel.

[0451] S1814. The keyboard and mouse virtualization capability module of the PC registers a keyboard and mouse event listener based on the instruction information for starting the keyboard and mouse service.

[0452] The PC's keyboard and mouse virtualization module registers a keyboard and mouse event listener to monitor keyboard and mouse events. For example, the PC's keyboard and mouse virtualization module can monitor keyboard presses and mouse movements, clicks, and drags.

[0453] S1815. The keyboard and mouse virtualization capability module of the PC sets keyboard and mouse event interception based on the instruction information for starting the keyboard and mouse service.

[0454] When the keyboard and mouse pass through, the keyboard and mouse events are intercepted on the PC side, and the response to the keyboard and mouse events is executed by the Android device. Therefore, the keyboard and mouse virtualization capability module of the PC sets the keyboard and mouse event interception to facilitate the interception of keyboard and mouse events after the keyboard and mouse pass through.

[0455] The embodiment of the present application does not limit the execution order of the above S1814 and S1815.

[0456] S1816, the keyboard and mouse virtualization capability module of the PC saves the processing result R 12 , where the processing result R 12 Including the results of whether the keyboard and mouse registration is successful and whether the keyboard and mouse event interception is set successfully.

[0457] Exemplarily, processing result B may include successful registration of the keyboard and mouse components and successful setting of keyboard and mouse event interception.

[0458] S1817: The keyboard and mouse virtualization capability module of the PC sends the processing result R to the keyboard and mouse service module of the PC. 12 Correspondingly, the keyboard and mouse service module of the PC receives the processing result R 12 .

[0459] S1818. The keyboard and mouse service module of the PC sends the processing result R to the keyboard and mouse service module of the Android device. 12 , correspondingly, the keyboard and mouse service module of the Android device receives the processing result R 12 .

[0460] S1819, the keyboard and mouse service module of the PC and the keyboard and mouse service module of the Android device are processed according to the result R 11 and the processing result R 12 , confirm whether the keyboard and mouse are connected successfully.

[0461] If both processing result A and processing result B are successful, the keyboard and mouse service module of the PC and the keyboard and mouse service module of the Android device can determine that the keyboard and mouse connection is successful. If at least one of the results in processing result A and processing result B is a failure, the keyboard and mouse service module of the PC and the keyboard and mouse service module of the Android device can determine that the keyboard and mouse connection is unsuccessful.

[0462] If the keyboard and mouse service module of the PC and the keyboard and mouse service module of the Android device are not successfully connected, both are restored to the initialization state, that is, the above registration process is re-executed.

[0463] The keyboard and mouse connection method provided in the embodiment of the present application performs keyboard and mouse connection between different devices during keyboard and mouse crossing, so as to prepare for subsequent keyboard and mouse crossing, which is conducive to the smooth progress of keyboard and mouse crossing.

[0464] After confirming that the keyboard and mouse are successfully connected through the above method 1800, the PC and the Android device can perform keyboard and mouse traversal.

[0465] Before introducing keyboard and mouse crossing, we first introduce a method for detecting the mouse position. Based on this method, the PC can monitor whether the mouse crosses.

[0466] For example, Figure 19 shows a schematic diagram of a method 1900 for detecting a mouse position. As shown in Figure 19 , the method 1900 may include the following steps:

[0467] S1901. The keyboard virtualization capability module of the PC obtains the current position of the mouse.

[0468] The current mouse position is the coordinates of the mouse on the screen. The PC's keyboard virtualization capability module can create a two-dimensional coordinate system on the screen, and the coordinates of the mouse in the two-dimensional coordinate system are the current mouse position. The current mouse position can be represented by (XRaw, YRaw), but the embodiments of the present application are not limited to this.

[0469] For example, FIG20 shows a schematic diagram of establishing a screen coordinate system. As shown in FIG20 , the screen OABC establishes a coordinate system with point O as the origin, the direction of OA as the positive direction of the X axis, and the direction of OC as the positive direction of the Y axis. The screen resolution is 1920*1080, the length of line segment OA is equal to the length of line segment CB, which is equal to 1920 pixels, and the length of line segment OC is equal to the length of line segment AB, which is equal to 1080 pixels. Then, in the coordinate system, the coordinates of point O are (0,0), the coordinates of point A are (1920,0), the coordinates of point B are (1920,1080), and the coordinates of point C are (0,1080). The mouse position is the coordinate of the mouse in the coordinate system. For example, the mouse position can be (1920,960).

[0470] S1902: When the keyboard virtualization capability module of the PC detects mouse movement, it obtains the mouse displacement.

[0471] Mouse displacement is the distance the mouse moves. Mouse displacement can be represented by (Lx, Ly). Lx and Ly can be positive or negative values, and this embodiment of the application does not limit this.

[0472] S1903. The keyboard virtualization capability module of the PC calculates the position of the mouse after the movement based on the mouse position and the mouse displacement.

[0473] The position of the mouse after it moves can be expressed by (curX, curY). The position of the mouse after it moves can be expressed by the following formula: curX = XRaw + Lx; curY = YRaw + Ly.

[0474] S1904: The keyboard virtualization capability module of the PC determines whether the position of the mouse after movement exceeds the screen.

[0475] In a scenario where the mouse moves left or right across the screen, if curX is less than the minimum X-axis value of the screen or greater than the maximum X-axis value of the screen, the PC's keyboard virtualization capability module can determine that the mouse is off-screen.

[0476] In a scenario where the mouse moves up and down the screen, if curY is less than the minimum value of the screen on the Y axis or greater than the maximum value of the screen on the Y axis, the PC's keyboard virtualization capability module can determine that the mouse is out of the screen.

[0477] If the position of the mouse after moving exceeds the screen, it can be determined that the mouse has crossed, that is, execute S1905. If the position of the mouse after moving does not exceed the screen, the position of the mouse after moving can be determined as the current position, and continue to obtain the mouse movement, that is, execute S1902.

[0478] S1905: If the position of the mouse after movement exceeds the screen, the keyboard virtualization capability module of the PC may determine that the mouse has crossed.

[0479] The PC can also determine the direction of the mouse movement. For example, if the mouse moves left and right on the screen, the direction can be left or right. If the mouse moves up and down on the screen, the direction can be up or down.

[0480] For example, in the example shown in FIG. 20 , if curX is less than 0 and the mouse is moving left, the PC can determine that the mouse is moving right. If curX is greater than 1920, the PC can determine that the mouse is moving right. If curY is less than 0 and the mouse is moving up, the PC's keyboard virtualization module can determine that the mouse is moving up. If curX is greater than 1080, the PC can determine that the mouse is moving down.

[0481] The mouse crossing detection method provided in the embodiment of the present application can determine whether the mouse is beyond the screen and the crossing direction through the mouse position and mouse displacement, which is conducive to improving the accuracy of detecting mouse crossing.

[0482] After the keyboard and mouse connection is successfully determined by the above method 1800, the PC and the Android device can perform keyboard and mouse traversal. The following describes the keyboard and mouse traversal method provided by the embodiment of the present application.

[0483] For example, Figure 21 shows a schematic flow chart of a keyboard and mouse traversal method 2100. This method 2100 can be applied to a communication system including a PC and an Android device, such as the aforementioned communication system 100, but the present invention is not limited thereto. The software architecture diagram of the PC can be shown in Figure 13, and the software architecture diagram of the Android device can be shown in Figure 14, but the present invention is not limited thereto.

[0484] The method 2100 may include the following steps:

[0485] S2101. The keyboard and mouse service module of the PC establishes a keyboard and mouse connection with the keyboard and mouse service module of the Android device.

[0486] This step can refer to the above-mentioned method 1800 and will not be repeated here.

[0487] S2102: If the keyboard and mouse virtualization capability module of the PC detects the user moving the mouse, and if it is detected that the keyboard and mouse do not cross, the keyboard and mouse virtualization capability module of the PC may respond to the detected keyboard and mouse event.

[0488] S2103. The keyboard and mouse virtualization capability module of the PC detects the user's operation of moving the mouse, and if the keyboard and mouse cross is detected.

[0489] The method for monitoring keyboard and mouse crossing can refer to the above method 1900 and will not be repeated here.

[0490] Optionally, after detecting the keyboard and mouse traversal, the keyboard and mouse virtualization capability module of the PC may save information such as the traversal device ID and whether it is the first traversal in the table tC.

[0491] S2104. The keyboard and mouse virtualization capability module of the PC obtains the mouse traversal position.

[0492] S2105. The keyboard and mouse virtualization capability module of the PC sends keyboard and mouse cross-talk indication information to the keyboard and mouse service module of the Android device. Correspondingly, the keyboard and mouse service module of the Android device receives the keyboard and mouse cross-talk indication information.

[0493] The keyboard and mouse virtualization capability module of the PC notifies the Android device of keyboard and mouse pass-through through the keyboard and mouse pass-through instruction information.

[0494] S2106. The keyboard and mouse service module of the Android device sends the keyboard and mouse pass-through indication information to the keyboard and mouse virtualization capability module of the PC of the Android device based on the keyboard and mouse pass-through indication information. Correspondingly, the keyboard and mouse virtualization capability module of the Android device receives the keyboard and mouse pass-through indication information.

[0495] S2107. The keyboard and mouse virtualization capability module of the Android device determines the mouse crossing position based on the indication information of the keyboard and mouse crossing.

[0496] If the screen resolutions of the Android device and PC are different, the mouse will appear in different positions on the screen when moving between the two devices. The keyboard and mouse virtualization module of the Android device can determine the position of the mouse on the Android device screen based on the position of the mouse on the PC screen.

[0497] S2108. The keyboard and mouse virtualization capability module of the Android device updates the cross-travel status based on the indication information of the keyboard and mouse cross-travel.

[0498] The keyboard and mouse virtualization capability module of the Android device can update the cross-travel status to cross-travel in progress based on the indication information of the keyboard and mouse cross-travel.

[0499] S2109. The keyboard and mouse virtualization capability module of the PC intercepts the monitored keyboard and mouse events.

[0500] When the mouse passes from the PC to the Android device, the PC's keyboard and mouse virtualization module intercepts the monitored keyboard and mouse events, making the PC unresponsive to mouse events.

[0501] S2110. The keyboard and mouse virtualization capability module of the PC packages the keyboard and mouse events to obtain packaged keyboard and mouse events.

[0502] The keyboard and mouse virtualization capability module of the PC can package the keyboard and mouse events through a preset protocol to obtain packaged keyboard and mouse events.

[0503] S2111. The keyboard and mouse virtualization capability module of the PC sends the packaged keyboard and mouse events to the keyboard and mouse virtualization capability module of the Android device. Correspondingly, the keyboard and mouse virtualization capability module of the Android device receives the packaged keyboard and mouse events.

[0504] Optionally, the keyboard and mouse virtualization capability module of the PC may further encrypt the packaged keyboard and mouse events before sending them to prevent the keyboard and mouse events from being leaked.

[0505] S2112. The keyboard and mouse virtualization capability module of the Android device parses the packaged keyboard and mouse events, obtains the keyboard and mouse events, and responds to the keyboard and mouse events.

[0506] When the mouse passes from the PC to the Android device, the Android device can parse the packaged keyboard and mouse events, obtain the keyboard and mouse events, and respond to the keyboard and mouse events. In this way, the PC and Android device can achieve keyboard and mouse pass-through.

[0507] S2113: The keyboard and mouse virtualization capability module of the Android device detects keyboard and mouse crossing.

[0508] When the keyboard and mouse are used on the Android device, the keyboard and mouse virtualization module of the Android device continuously monitors the keyboard and mouse crossing.

[0509] After the mouse has been passed to the Android device, it can be passed back. In the scenario where the mouse is passed back to the PC, the keyboard and mouse pass-through operation can be understood as the keyboard and mouse pass-back operation, that is, passing from the Android device back to the PC. The following will continue to explain.

[0510] S2114. The keyboard and mouse virtualization capability module of the Android device sends a message to the keyboard and mouse service module of the PC to end the cross-travel. Correspondingly, the keyboard and mouse service module of the PC receives the message to end the cross-travel.

[0511] The message of ending the cross-platform operation means stopping the keyboard and mouse from crossing from the PC to the Android device.

[0512] S2115. The keyboard and mouse service module of the PC sends a message of ending the crossing-through to the keyboard and mouse virtualization capability module of the PC based on the message of ending the crossing-through. Correspondingly, the keyboard and mouse virtualization capability module of the PC receives the message of ending the crossing-through.

[0513] S2116. The keyboard and mouse virtualization capability module of the PC stops intercepting the monitored keyboard and mouse events based on the message of ending the traversal.

[0514] The keyboard and mouse are transmitted from the Android device back to the PC, and the PC responds to the keyboard and mouse operations. Therefore, the keyboard and mouse virtualization capability module of the PC can stop intercepting the monitored keyboard and mouse events.

[0515] S2117. The keyboard and mouse virtualization capability module of the PC updates the travel status based on the message of ending the travel.

[0516] The keyboard and mouse virtualization capability module of the PC can update the traversal state from traversing to not traversing based on the message of ending traversal.

[0517] The embodiment of the present application does not limit the execution order of S2116 and S2117.

[0518] S2118. The keyboard and mouse virtualization capability module of the PC sends a setting completion message to the keyboard and mouse virtualization capability module of the Android device. Correspondingly, the keyboard and mouse virtualization capability module of the Android device receives the setting completion message.

[0519] The setup completion message is used to indicate that interception of monitored keyboard and mouse events has stopped and the traversal status has been updated.

[0520] S2119. The keyboard and mouse virtualization capability module of the Android device updates the traversal status based on the setting completion message.

[0521] When the keyboard and mouse are transferred to the Android device, the transfer status is in progress. When the keyboard and mouse are transferred back to the PC, the keyboard and mouse virtualization capability module of the Android device can update the transfer status from in progress to not transferred based on the setting completion message.

[0522] The keyboard and mouse crossing method provided in the embodiment of the present application, when the keyboard and mouse achieve cross-device crossing, the mouse crossing position can be adjusted based on different devices. After the mouse crossing is successful, the keyboard and mouse events are intercepted at the PC and responded to on the Android device side, thereby achieving cross-device crossing of the mouse.

[0523] In the above method 2100, when the keyboard and mouse are passed from the PC to the Android device, due to the different resolutions of the PC and the Android device, the Android device can execute S2107 to determine the mouse crossing position, that is, the position on the Android device screen. For example, in the example shown in Figure 2 above, as shown in Figure 2 b, the mouse passes from the computer 101 to the initial display position of the mobile phone 102.

[0524] An embodiment of the present application provides a method for determining the initial position of a mouse crossing, which can calculate the position of the mouse on the Android device screen through the position of the mouse on the PC screen, the screen resolution of the PC, and the screen resolution of the Android device.

[0525] For example, Figure 22 shows a schematic flow chart of a method 2200 for determining the initial position of a mouse traversal. This method 2200 can be applied to a communication system including a PC and an Android device, such as the aforementioned communication system 100, but embodiments of the present application are not limited thereto. The software architecture diagram for the PC can be shown in Figure 13, and the software architecture diagram for the Android device can be shown in Figure 14, but embodiments of the present application are not limited thereto.

[0526] The method 2200 may include the following steps:

[0527] S2201. The keyboard and mouse service module of the PC establishes a keyboard and mouse connection with the keyboard and mouse service module of the Android device.

[0528] This step is the same as the above-mentioned S2101.

[0529] S2202. The keyboard and mouse service module of the Android device obtains the screen resolution.

[0530] It is understandable that the screen resolution is that of the Android device.

[0531] S2203. The keyboard and mouse service module of the Android device sends the screen resolution to the keyboard and mouse service module of the PC. Correspondingly, the keyboard and mouse service module of the PC receives the screen resolution.

[0532] S2204 : The keyboard and mouse service module of the PC sends the screen resolution to the keyboard and mouse virtualization capability module of the PC based on the screen resolution. Correspondingly, the conversion module of the PC receives the screen resolution.

[0533] S2205. The keyboard and mouse virtualization capability module of the PC calculates the mouse sensitivity based on the screen resolution and the screen resolution of the PC.

[0534] Mouse sensitivity can be the ratio of the screen resolution of the Android device to the screen resolution of the PC.

[0535] For example, the sensitivity conversion rate s can be calculated using the following formula:

[0536] Among them, R1 is the screen resolution of the Android device, and R2 is the screen resolution of the PC.

[0537] If the screen resolution of the Android device R1=R 1x *R 1y , where R 1x and R 1y is the side length of the Android device screen, and the PC screen resolution R2=R 2x *R 2y , where R 2x and R 2y is the side length of the PC screen, the sensitivity conversion rate s includes s x and s y , where s x and s y It can be expressed by the following formula:

[0538] It should be noted that the above S2202 to S2205 are steps that can be executed after the Android device establishes a mouse connection with the PC.

[0539] S2206: The keyboard and mouse virtualization capability module of the PC detects keyboard and mouse crossing.

[0540] The keyboard and mouse virtualization capability module of the PC monitors the keyboard and mouse crossing through the above method 1900.

[0541] S2207: The keyboard and mouse virtualization capability module of the PC obtains the mouse traversal position.

[0542] S2208. The keyboard and mouse virtualization capability module of the PC converts the mouse traversal position (which may be referred to as position for short) based on the mouse sensitivity to obtain a converted position.

[0543] If the mouse's crossing position on the PC screen is (x1, y1) and the crossing direction is left crossing or right crossing, the PC's conversion module can be based on s y Change the transformation of y1, that is, y1*s y The horizontal axis value is the preset value. The preset value is the minimum or maximum value of the screen side length on the X axis.

[0544] For example, in the example shown in FIG. 20 , if the crossing position is (1920, 960) and the crossing direction is right crossing, the position after conversion can be (0, 960*s y ). If the crossing position is (0,960), and the crossing direction is right crossing, and the screen resolution of the Android device is 2000*1000, the converted position can be (2000, 960*s y ).in,

[0545] If the mouse's crossing position on the PC screen is (x1, y1) and the crossing direction is up or down, the PC's conversion module can be based on s x Change the transformation of x1, that is, x1*s x The vertical axis value is the preset value. The preset value is the minimum or maximum value of the screen side length on the X axis.

[0546] S2209 . The keyboard and mouse virtualization capability module of the PC sends the converted position to the keyboard and mouse virtualization capability module of the Android device. Correspondingly, the keyboard and mouse virtualization capability module of the Android device receives the converted position.

[0547] S2210. The keyboard and mouse virtualization capability module of the Android device sends instruction information for displaying the mouse to the UI module. Correspondingly, the UI module receives the instruction information for displaying the mouse.

[0548] S2211. The UI module displays the mouse pointer at the converted position based on the indication information of the display mouse.

[0549] The method for determining the initial position of the mouse crossing provided in the embodiment of the present application converts the mouse crossing position through the sensitivity conversion rate, so that when the mouse crosses to the Android device, it can adapt to the screen resolution of the Android device, which is conducive to controlling the Android device with the mouse.

[0550] If the resolutions of the PC and the Android device are different, in addition to affecting the mouse traversal position, it will also affect the mouse movement speed. Among them, the mouse movement speed can also be called mouse sensitivity, which is not limited in the embodiments of the present application.

[0551] The present application provides a method for determining mouse movement speed, which can calculate the mouse movement speed on an Android device by using the mouse movement speed on a PC and the sensitivity conversion rate. For example, in the example shown in FIG2 above, FIG2 b shows the movement speed of the mouse from the computer 101 to the mobile phone 102.

[0552] The method for determining the mouse movement speed provided in the embodiment of the present application can be executed between S2108 and S2109 above.

[0553] For example, FIG23 shows a schematic flow chart of a method 2300 for determining mouse movement speed. This method 2300 can be applied to a communication system including a PC and an Android device, such as the aforementioned communication system 100, but embodiments of the present application are not limited thereto. A software architecture diagram for a PC can be shown in FIG13 , and a software architecture diagram for an Android device can be shown in FIG14 , but embodiments of the present application are not limited thereto.

[0554] The method 2300 may include the following steps:

[0555] S2301, the keyboard and mouse virtualization capability module of the PC obtains the mouse movement speed V A .

[0556] The movement speed of the mouse is the displacement per unit time, which can also be understood as the number of pixels moved per unit time. The unit time can be preset, for example, 3 milliseconds, 4 milliseconds, or 20 milliseconds, which is not limited in this embodiment of the present application. The movement speed can be represented by (Vx, Vy), but this embodiment of the present application is not limited to this.

[0557] S2302: The keyboard and mouse virtualization capability module of the PC converts the moving speed A to obtain the moving speed V. B , correspondingly, the conversion module of the PC receives the moving speed V B .

[0558] The PC's keyboard and mouse virtualization capability module can be based on the sensitivity conversion rate s x and s yMovement speed V A (V x ,V y ) can be converted to obtain the moving speed V B (V x *s x ,V y *s y ). The sensitivity conversion rate can be obtained through the above-mentioned S2201 to S2204, which will not be repeated here.

[0559] For example, the screen resolution of the PC is R2=R 2x *R 2y =3000*2000, the screen resolution of Android devices is R1=R 1x *R 1y =2000*1000, then If the moving speed V A (V x ,V y )(V x ,V y )=(3,2), then That is, the moving speed V B (2,1).

[0560] S2303, the PC keyboard and mouse virtualization capability module controls the movement speed V B Perform packing and obtain the moving speed B after packing.

[0561] S2304: The keyboard and mouse virtualization capability module of the PC sends the packaged movement speed V to the keyboard and mouse virtualization capability module of the Android device. B .

[0562] The PC keyboard and mouse virtualization module can control the movement speed V through the preset protocol. B Pack and get the moving speed V after packing B . For moving speed V B Packing is beneficial to the movement speed V B Keep it confidential and facilitate cross-device transmission speed V B .

[0563] S2305. The keyboard and mouse virtualization capability module of the Android device sends instruction information for displaying the mouse to the UI module. Correspondingly, the UI module receives the instruction information for displaying the mouse.

[0564] Display the mouse pointer information to instruct the UI module to move at a speed of V B Move the mouse.

[0565] S2306: The UI module can move the mouse at a speed V based on the indication information of the display mouse.B Move the mouse.

[0566] The mouse movement speed determination method provided in the embodiment of the present application converts the mouse movement speed through the sensitivity conversion rate, so that when the mouse crosses to an Android device, the mouse movement speed is adapted to the screen resolution of the Android device, which is conducive to controlling the Android device through the mouse and improving the usability of the mouse after crossing the device.

[0567] In the above method 2100, S2109, the keyboard and mouse virtualization capability module of the PC intercepts the monitored keyboard and mouse events. The embodiment of the present application provides a specific implementation method.

[0568] For example, Figure 24 shows a schematic flow chart of a method 2400 for intercepting keyboard and mouse events on a PC. This method can be performed by a PC keyboard and mouse virtualization module and a PC keyboard and mouse event detection module. The PC keyboard and mouse virtualization module includes a hidden window module, a keyboard and mouse management module, an edge computing module, and a keyboard and mouse interception module.

[0569] As shown in FIG. 24 , the method 2400 may include the following steps:

[0570] S2401. The keyboard and mouse management module sends instruction information for creating a hidden window to the hidden window module. Correspondingly, the hidden window module receives the instruction information for creating a hidden window.

[0571] The instruction information for creating a hidden window may be used to instruct the hidden window module to hide the window.

[0572] S2402: The hidden window module creates a hidden window based on the instruction information for creating a hidden window.

[0573] A hidden window can be thought of as a window running in the background of a PC. It receives keyboard and mouse events. Keyboard and mouse events are a type of Windows message. In Windows programming, when each Windows application begins executing, the system creates a message queue for that application. This message queue is used to store information about windows created by that application.

[0574] For example, when the user presses the right mouse button, the PC generates a WM_RBUTTONDOWN message. The system automatically places this message into the message queue of the application to which the hidden window belongs, waiting for the application to terminate. Windows sequentially places the generated messages into the message queue, and the application continuously reads messages from the message queue through a message loop and responds. After the hidden window module successfully creates the hidden window, it receives a WM_INPUT message.

[0575] S2403: In response to the user's operation of moving the mouse, the hiding window module obtains the mouse position S through the hidden window. A .

[0576] The hidden window module can read the mouse position in the hidden window A .

[0577] S2404, the hidden window module sends the mouse position S to the keyboard and mouse management module A , correspondingly, the keyboard and mouse management module receives the mouse position S A .

[0578] S2405, the keyboard and mouse management module is based on the mouse position S A , sends edge detection indication information to the edge computing module, and correspondingly, the edge computing module receives the edge detection indication information.

[0579] The edge detection indication information is used to instruct the edge computing module to determine whether the mouse is at the edge of the screen.

[0580] S2406. The edge computing module detects that the mouse goes beyond the edge of the screen based on the indication information of the edge detection.

[0581] The edge computing module is based on the mouse position S A , determining whether the mouse has exceeded the edge of the screen. Specific implementation methods can refer to the above method 1900.

[0582] S2407. The edge computing module sends instruction information for setting the hook to the keyboard and mouse interception module. Correspondingly, the keyboard and mouse interception module receives the instruction information for setting the hook.

[0583] The indication information of the interception event is used to instruct the keyboard and mouse interception module to set up keyboard hooks, mouse hooks, and shortcut hooks. Among them, the hook is a platform of the Windows message processing mechanism. Applications can set up subroutines on it to monitor certain messages of a specified window, and the monitored window can be created by other processes. When a message arrives, it is processed before the target window processing function. The hook mechanism allows applications to intercept and process window messages or specific events. The hook is actually a program segment that processes messages, which is hung into the system through system calls. Whenever a specific message is sent, before it reaches the destination window, the hook program will capture the message first, that is, the hook function will get control first. At this time, the hook function can process (change) the message, or continue to pass the message without processing it, or force the message transmission to end.

[0584] The keyboard hook is used to obtain keyboard input operations, for example, the keyboard hook is used to obtain the user's keyboard input "Unprocessed items". The mouse hook is used to obtain mouse control operations, for example, the mouse hook is used to obtain the user's mouse click or drag operations. The shortcut key hook is used to obtain the user's shortcut key operations, for example, the shortcut key hook is used to obtain the user's Ctrl+C, Ctrl+V, ATL+TAB, or Ctrl+Alt+DELETE operations.

[0585] S2408. The keyboard and mouse interception module sets the keyboard hook, the mouse hook, and the shortcut key hook based on the instruction information for setting the hook.

[0586] S2409: The keyboard and mouse interception module sends the indication information of the keyboard and mouse event reporting to the keyboard and mouse event detection module. Correspondingly, the keyboard and mouse event detection module receives the indication information of the keyboard and mouse event reporting.

[0587] The indication information for reporting keyboard and mouse events is used to instruct the keyboard and mouse event detection module to report keyboard and mouse events.

[0588] S2410. The keyboard and mouse event detection module may report the acquired keyboard and mouse event to the keyboard and mouse interception module based on the indication information of the keyboard and mouse event report. Correspondingly, the keyboard and mouse interception module receives the keyboard and mouse event.

[0589] S2411. The keyboard and mouse interception module sends keyboard and mouse events to the edge computing module. Correspondingly, the edge computing module receives the keyboard and mouse events.

[0590] S2412. The edge computing module determines the keyboard and mouse crossing based on the keyboard and mouse events.

[0591] S2413. The edge computing module sends a notification message of the interception event to the keyboard and mouse management module. Correspondingly, the keyboard and mouse management module receives the notification message of the interception event.

[0592] The notification message of the interception event is used to instruct the keyboard and mouse management module to intercept the keyboard and mouse event.

[0593] S2414. The keyboard and mouse management module intercepts the keyboard and mouse events based on the notification message of the interception event.

[0594] After the keyboard and mouse management module intercepts the keyboard and mouse events, the PC executes the above S2110 to S2112, that is, the keyboard and mouse events are packaged and transmitted to the Android device.

[0595] S2415. The keyboard and mouse management module sends instruction information for stopping reporting of keyboard and mouse events to the keyboard and mouse interception module. Correspondingly, the keyboard and mouse interception module receives the instruction information for stopping reporting of keyboard and mouse events.

[0596] The instruction information for stopping reporting of keyboard and mouse events is used to instruct the keyboard and mouse event detection module to stop reporting keyboard and mouse events.

[0597] S2416. The keyboard and mouse interception module sends instruction information for stopping reporting of keyboard and mouse events to the keyboard and mouse event detection module. Correspondingly, the keyboard and mouse event detection module receives the instruction information for stopping reporting of keyboard and mouse events.

[0598] S2417: In response to the user moving the mouse, the hidden window module obtains the mouse position S B .

[0599] Mouse position S B With the above mouse position S A different.

[0600] S2418, the hidden window module sends the mouse position S to the keyboard and mouse management module B , correspondingly, the keyboard and mouse management module receives the mouse position S B .

[0601] S2419, keyboard and mouse management module based on the mouse position S B , sends edge detection indication information to the edge computing module, and correspondingly, the edge computing module receives the edge detection indication information.

[0602] S2420. The edge computing module monitors the mouse entering the edge of the screen based on the indication information of the edge detection.

[0603] The mouse enters the edge of the screen, that is, the mouse passes from the Android device screen back to the PC screen.

[0604] For example, in the example shown in Figure 2 above, the user moves the mouse from computer 101 to mobile phone 102. After completing the editing of the memo application, the user can continue to move the mouse to computer 101 to operate computer 101. When the user moves the mouse from the screen back to the PC screen, in response to the user's operation, the edge computing module can determine that the mouse has entered the edge of the screen.

[0605] S2421. The edge computing module sends an instruction to cancel the hook to the keyboard and mouse interception module. Correspondingly, the keyboard and mouse interception module receives the instruction to cancel the hook.

[0606] The instruction information for canceling the hook is used to instruct the keyboard and mouse interception module to cancel setting the hook.

[0607] S2422. The keyboard and mouse interception module cancels the keyboard hook, the mouse hook, and the shortcut key hook based on the instruction information for canceling the hook.

[0608] S2423. The keyboard and mouse interception module sends indication information of keyboard and mouse event reporting to the keyboard and mouse event detection module. Correspondingly, the keyboard and mouse event detection module receives the indication information of keyboard and mouse event reporting and reports the keyboard and mouse event based on the indication information of keyboard and mouse event reporting.

[0609] The method for intercepting keyboard and mouse events provided in an embodiment of the present application uses a keyboard hook, a mouse hook, and a shortcut key hook on the PC side to intercept the keyboard and mouse events when the keyboard and mouse pass through to an Android device, and responds to the keyboard and mouse events on the Android device side. When the keyboard and mouse pass back to the PC, the keyboard hook, the mouse hook, and the shortcut key hook are canceled on the PC side to respond to the keyboard and mouse events, which is conducive to realizing cross-device mouse crossing.

[0610] When the keyboard and mouse are transferred between different devices, the user may not be able to see the mouse on the screen, or the user may not need the keyboard and mouse to operate on the Android device and want to control the PC with the keyboard and mouse. For example, in the example shown in Figure 3 above, the user can trigger the Esc operation to display the mouse pointer 1011 in the center of the computer 101 screen.

[0611] To meet user needs, this embodiment of the present application provides a shortcut key for forcibly reclaiming the keyboard and mouse. When the PC detects that the user has triggered the shortcut key, the mouse cursor is displayed on the PC screen. The mouse cursor can be positioned in the center of the PC screen, but this embodiment of the present application is not limited thereto.

[0612] For example, Figure 25 shows a schematic flow chart of a method 2500 for forcibly reclaiming a keyboard and mouse. This method can be performed by a PC's keyboard and mouse virtualization capability module and a PC's keyboard and mouse event detection module. The PC's keyboard and mouse virtualization capability module includes a hidden window module, a keyboard and mouse management module, an edge computing module, and a keyboard and mouse interception module.

[0613] As shown in FIG. 25 , the method 2500 may include the following steps:

[0614] S2501. The keyboard and mouse event detection module sends a preset shortcut key event to the keyboard and mouse interceptor.

[0615] The keyboard and mouse event detection module receives the indication information of the keyboard and mouse event report and can report the keyboard and mouse event. When a preset shortcut key event is detected, the keyboard and mouse event detection module sends the preset shortcut key event to the keyboard and mouse interceptor.

[0616] The preset shortcut key is also the shortcut key for forced recapture. The shortcut key for forced recapture can be Esc or Ctrl+Alt+DELETE, which is not limited in the embodiment of the present application.

[0617] S2502. The keyboard and mouse interception module sends a notification message of the preset shortcut key event to the edge computing module based on the preset shortcut key event. Correspondingly, the edge computing module receives the notification message of the preset shortcut key event.

[0618] The notification message of the preset shortcut key event is used to notify the edge computing module that an operation of forcibly reclaiming the mouse has been received.

[0619] S2503. Based on the notification message of the preset shortcut key event, the edge computing module sends a notification message of reclaiming the mouse to the keyboard and mouse management module in the case of keyboard and mouse crossing. Correspondingly, the keyboard and mouse management module receives the notification message of reclaiming the mouse.

[0620] The notification message of reclaiming the mouse is used to notify the keyboard and mouse management module to forcibly reclaim the mouse.

[0621] S2504: The keyboard and mouse management module forcibly reclaims the mouse based on the notification message for reclaiming the mouse.

[0622] After the keyboard and mouse management module forcibly recaptures the mouse, the mouse can be displayed in the middle of the PC screen. It is understandable that the display position after forcibly recaptures the mouse is preset and can be the middle of the screen, but the embodiment of the present application is not limited thereto.

[0623] S2505. The edge computing module sends an instruction to cancel the hook to the keyboard and mouse interception module based on the notification message of the preset shortcut key event. Correspondingly, the keyboard and mouse interception module receives the instruction to cancel the hook.

[0624] The instruction information for canceling the hook is used to instruct the keyboard and mouse interception module to cancel setting the hook.

[0625] The embodiment of the present application does not limit the execution order of the above S2504 and S2505.

[0626] S2506: The keyboard and mouse interception module cancels the keyboard hook, the mouse hook, and the shortcut key hook based on the instruction information for canceling the hook.

[0627] S2507. The keyboard and mouse interception module sends instruction information for stopping reporting of keyboard and mouse events to the keyboard and mouse event detection module. Correspondingly, the keyboard and mouse event detection module receives the instruction information for stopping reporting of keyboard and mouse events and no longer sends keyboard and mouse events to the keyboard and mouse interception module.

[0628] The method for forced recapture of the keyboard and mouse provided in the embodiment of the present application can forcibly recapture the mouse when an operation of forced recapture of the mouse is monitored, thereby meeting the user's need for forced recapture of the mouse. Moreover, forced recapture of the mouse by using a shortcut key is more efficient.

[0629] Optionally, when a shortcut for forcibly reclaiming the mouse is set, the above-mentioned step S2212, in which the keyboard and mouse management module intercepts the keyboard and mouse event based on the notification message of the interception event, may include: the keyboard and mouse management module determining whether the keyboard and mouse event is a shortcut event for forcibly reclaiming the mouse, and if not, intercepting it. If so, forcibly reclaiming the mouse.

[0630] In a scenario where the PC and the Android device are connected, if the position of the Android device relative to the PC changes, the PC can update the traversal direction in table tC according to the changed position of the Android device.

[0631] The embodiments of the present application provide two methods for detecting the location change of an Android device: automatic and manual.

[0632] In one possible implementation, if a user adjusts the position of a connected device, the PC detects the change in position using ultrasound and updates the crossing direction in table tC based on the changed position. For example, in the scenario shown in FIG10 , if computer 101 detects that phone 102 has moved from the left side of the computer to the right side of the computer 101, the crossing direction from computer 101 to phone 102 can be updated to right crossing.

[0633] For example, Figure 26 shows a schematic flow chart of a method 2600 for adjusting a traversal direction. This method 2600 can be applied to a communication system including a PC and an Android device connected, such as the aforementioned communication system 100, but embodiments of the present application are not limited thereto. The software architecture diagram of the PC can be shown in Figure 13, and the software architecture diagram of the Android device can be shown in Figure 14, but embodiments of the present application are not limited thereto. In this method, the Android device can also be referred to as a connected device.

[0634] The method 2600 may include the following steps:

[0635] S2601: When the smart capability is enabled, the keyboard and mouse service module of the PC detects a change in the direction information of the connected device through table tC.

[0636] Table tC includes the direction information (i.e., coordinate direction) of the connected device. If the direction information of the connected device changes, the coordinate direction in table tC will also change. The keyboard and mouse service module of the PC detects the change in the coordinate direction in table tC and determines that the direction information of the connected device has changed.

[0637] The number of connected devices can be 1 or 2, which is not limited in the embodiment of the present application.

[0638] The direction information of the connected device may change in the following situations:

[0639] 1) When the number of connected devices is 1, the coordinate direction changes to the opposite direction or the middle.

[0640] Exemplarily, the coordinate direction of the connected device is left, and when the coordinate direction changes, the coordinate direction changes to right or middle.

[0641] 2) When the number of connected devices is 2, the coordinates of only one connected device are changed, and the coordinate direction is changed to the opposite direction or the middle direction.

[0642] For example, the number of connected devices is 2, and the two connected devices are connected device 1 and connected device 2. The coordinate direction of connected device 1 is left, and the coordinate direction of connected device 2 is right. The coordinate direction of connected device 1 changes to the right or the center. It is understood that whether the coordinate direction changes to the right or the center is determined by the user.

[0643] 3) When the number of connected devices is 2, the coordinates of both connected devices change, and the coordinate directions change to opposite directions or to the middle, or the coordinate direction of one connected device changes to the opposite direction, and the coordinate direction of the other connected device changes to the middle.

[0644] For example, the number of connected devices is two, and the two connected devices are connected device 1 and connected device 2. The coordinate direction of connected device 1 is left, and the coordinate direction of connected device 2 is right. The coordinate directions of connected device 1 and connected device 2 can both change, with the coordinate direction of connected device 1 changing to the right and the coordinate direction of connected device 2 changing to the left. Alternatively, the coordinate direction of connected device 1 and the coordinate direction of connected device 2 can both change to the center. Alternatively, the coordinate direction of connected device 1 can change to the right, and the coordinate direction of connected device 2 can change to the center.

[0645] S2602: If the change in direction information meets the requirement of a crossing direction change, the keyboard and mouse service module of the PC sends information indicating direction adjustment to the keyboard and mouse virtualization capability module of the PC. Correspondingly, the keyboard and mouse virtualization capability module of the PC receives the information indicating direction adjustment.

[0646] Not all changes in direction information will result in an adjustment to the crossing direction. The situations listed above may include the following situations where changes in direction information qualify as changes in the crossing direction:

[0647] 1) When the number of connected devices is 1, the coordinate direction changes to the opposite direction.

[0648] 2) When the number of connected devices is 2, the coordinates of both connected devices are changed, and the coordinate directions are changed to opposite directions.

[0649] The information indicating the direction adjustment may be used to instruct the keyboard and mouse virtualization capability module of the PC to adjust the direction of travel.

[0650] S2603: The keyboard and mouse virtualization capability module of the PC updates the traversal direction based on the information indicating the direction adjustment.

[0651] The keyboard and mouse virtualization capability module of the PC can update the crossing direction to the opposite direction.

[0652] In particular, if the coordinate direction of the connected device changes to the middle, the crossing direction can be changed to the default direction or the direction of the last crossing. The default direction can be right crossing or left crossing, which is not limited in the embodiment of the present application.

[0653] For example, if the number of connected devices is 1, the coordinate direction is changed to the middle, and the default direction is right-crossing, the PC's keyboard and mouse virtualization capability module can change the crossing direction to the default direction. It is understood that if the crossing direction of the connected device was right-crossing before the coordinate direction was changed, the crossing direction will remain right-crossing when the coordinate direction is changed to the middle.

[0654] For example, if the number of connected devices is two and the coordinate directions of both connected devices are changed to the middle, the PC's keyboard and mouse virtualization capability module can use the tC table to determine whether the two connected devices are crossing for the first time. If so, the crossing direction can be updated to the default crossing direction. If not, the crossing direction can be updated to the direction of the last crossing.

[0655] S2604. The keyboard and mouse service module of the PC sends information indicating direction adjustment to the keyboard and mouse service module of the connected device. Correspondingly, the keyboard and mouse service module of the Android device receives the information indicating direction adjustment.

[0656] The traversal direction of the connected device will also change. The keyboard and mouse service module of the PC can instruct the keyboard and mouse service module of the connected device to update the traversal direction through the information indicating the direction adjustment.

[0657] S2605: The keyboard and mouse service module of the connected device updates the traversing direction based on the information indicating the direction adjustment.

[0658] The keyboard and mouse service module of the connected device can update the traversal direction to the opposite direction.

[0659] S2606. The keyboard and mouse service module of the connected device sends an update success message to the keyboard and mouse service module of the PC. Correspondingly, the keyboard and mouse service module of the PC receives the update success message.

[0660] The update success information can be used to inform the keyboard and mouse service module of the PC that the traversal direction has been updated. Subsequently, the PC and the connected device can implement keyboard and mouse traversal through the new traversal direction.

[0661] S2607. The keyboard and mouse service module of the PC updates table tC based on the update success information.

[0662] After the keyboard and mouse service module of the PC and the keyboard and mouse service module of the connected device both update the traversal direction, the keyboard and mouse service module of the PC may update the traversal direction in table tC.

[0663] The method for adjusting the crossing direction provided in the embodiment of the present application can automatically detect the change in the direction of the connected device, and change the crossing direction when the conditions for the change in the crossing direction are met. This is conducive to enabling the PC to change the crossing direction according to the change in the direction of the connected device, which is more in line with the user's usage habits and improves the user experience.

[0664] In one possible implementation, a user adjusts the position of a connected device in a visual interface provided by a PC. The PC detects the change in the device's position in the visual interface and updates the traversal direction in table tC based on the changed position. In the example shown in FIG9 , if the user adjusts the tablet from the left side of the computer to the right side of the computer in the visual interface, the computer 101 can change the traversal direction from the computer to the tablet to right traversal in response to the user's operation.

[0665] The embodiment of the present application provides a specific implementation method.

[0666] For example, Figure 27 shows a schematic flow chart of another method 2700 for adjusting the traversal direction. This method 2700 can be applied to a communication system including a PC and an Android device, such as the aforementioned communication system 100, but embodiments of the present application are not limited thereto. The software architecture diagram for the PC can be shown in Figure 13, and the software architecture diagram for the Android device can be shown in Figure 14, but embodiments of the present application are not limited thereto.

[0667] The method 2700 may include the following steps:

[0668] S2701: When the smart capability is turned off, the keyboard and mouse service module of the PC detects the user's operation of opening the visual interface.

[0669] The visual interface allows users to view the connection status of the PC and other devices, select devices to connect, and adjust the position of connected devices.

[0670] S2702: The keyboard and mouse service module of the PC displays connected devices and online devices on the visual interface in response to the user opening the visual interface.

[0671] The keyboard and mouse service module of the PC can display the connected devices on the visual interface through the above table tC, and can display the online devices on the visual interface through the above table tA.

[0672] S2703: The keyboard and mouse service module of the PC detects that the user changes the direction of the connected device.

[0673] For example, in the example shown in FIG7 above, the user drags the connected device tablet from the left side of the computer to the right side of the tablet. The keyboard and mouse service module of the PC detects the user's operation and can determine that the left direction of the connected device tablet changes from the left side to the right side.

[0674] The keyboard and mouse service module of the PC detects the user's operation of changing the direction of the connected device, and in response to the operation, the above S2602 to S2607 can be executed. The specific implementation method is the same and will not be repeated here.

[0675] The method for adjusting the crossing direction provided in the embodiment of the present application can determine that the direction information of the connected device has changed by detecting the user's operation to change the direction of the connected device, and change the crossing direction when the conditions for changing the crossing direction are met. This is conducive to enabling the PC to change the crossing direction according to the user's operation to conform to the user's usage habits and improve the user experience.

[0676] The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0677] The above describes in detail the method provided by the embodiment of the present application in combination with Figures 1 to 27. The following describes in detail the terminal device provided by the embodiment of the present application in combination with Figures 28 and 29.

[0678] 28 shows a schematic diagram of a terminal device 2800 provided in an embodiment of the present application. The terminal device 2800 includes: a display module 2810 and a processing module 2820 .

[0679] Among them, the display module 2810 is used to: display the cursor corresponding to the input device on the first screen; the processing module 2820 is used to: detect a first operation acting on the input device, the first operation is used to drag the cursor to determine the first direction as the connection direction; in response to the first operation, establish a connection with the second device located in the first direction of the first device; detect a second operation acting on the input device, the second operation is used to drag the cursor displayed on the first screen in the first direction; in response to the second operation, control the cursor to move out of the first screen and enter the second screen of the second device for display; wherein, in response to a third operation acting on the input device, the second device executes a control instruction corresponding to the third operation, and the third operation is used to input a control instruction on the second device.

[0680] The terminal device 2800 may be a PC, and may execute the various methods provided in the above-mentioned embodiments of the present application.

[0681] It should be understood that the terminal device 2800 here is embodied in the form of a functional module. The term "module" here can refer to an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor or a group processor, etc.) and a memory for executing one or more software or firmware programs, a combined logic circuit and / or other suitable components that support the described functions. In an optional example, those skilled in the art will understand that the terminal device 2800 can be specifically the PC in the above-mentioned method embodiment, or the functions of the PC in the above-mentioned method embodiment can be integrated into the terminal device 2800, and the terminal device 2800 can be used to execute the various processes and / or steps corresponding to the PC in the above-mentioned method embodiment. To avoid repetition, they will not be described here.

[0682] The terminal device 2800 has the function of implementing the corresponding steps performed by the PC in the above method embodiment; the above functions can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0683] In an embodiment of the present application, the terminal device 2800 in FIG. 28 may also be a chip or a chip system, such as a system on chip (SoC).

[0684] Figure 29 is a schematic block diagram of another terminal device 2900 provided in an embodiment of the present application. The terminal device 2900 includes a processor 2910, a transceiver 2920, and a memory 2930. The processor 2910, the transceiver 2920, and the memory 2930 communicate with each other via an internal connection path. The memory 2930 is used to store instructions, and the processor 2920 is used to execute the instructions stored in the memory 2930 to control the transceiver 2920 to send and / or receive signals.

[0685] It should be understood that the terminal device 2900 can be specifically the PC in the above-mentioned method embodiment, or the functions of the PC in the above-mentioned method embodiment can be integrated into the terminal device 2900, and the terminal device 2900 can be used to execute the various steps and / or processes corresponding to the PC in the above-mentioned method embodiment. Optionally, the memory 2930 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store device type information. The processor 2910 can be used to execute instructions stored in the memory, and when the processor executes the instructions, the processor can execute the various steps and / or processes corresponding to the terminal device in the above-mentioned method embodiment.

[0686] It should be understood that in the embodiment of the present application, the processor 2910 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0687] During implementation, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor executes the instructions in the memory, and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0688] The present application also provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to implement the method corresponding to the PC in the above method embodiment.

[0689] The present application also provides a chip system, which is used to support the PC in the above method embodiment to implement the functions shown in the embodiment of the present application.

[0690] The present application also provides a computer program product, which includes a computer program (also referred to as code or instruction). When the computer program runs on a computer, the computer can execute the method corresponding to the PC shown in the above method embodiment.

[0691] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0692] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0693] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0694] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0695] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0696] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0697] The above description is merely a specific embodiment of the present application, but the scope of protection of the embodiments of the present application is not limited thereto. Any person skilled in the art can easily conceive of changes or substitutions within the technical scope disclosed in the embodiments of the present application, and such changes or substitutions should be included in the scope of protection of the embodiments of the present application. Therefore, the scope of protection of the embodiments of the present application should be based on the scope of protection of the claims.

Claims

1. A keyboard and mouse traversal method, characterized in that: Applied to a first device, the first device is connected to an input device for inputting control instructions, and the first device includes a first screen; The method further comprises: The first device displays a cursor corresponding to the input device on the first screen; The first device detects a first operation applied to the input device, where the first operation is to drag the cursor to determine a first direction as a connection direction; In response to the first operation, the first device establishes a connection with a second device located in the first direction of the first device; The first device detects a second operation applied to the input device, where the second operation is used to drag the cursor displayed on the first screen toward the first direction; In response to the second operation, the first device controls the cursor to move out of the first screen and enter the second screen of the second device for display; wherein, in response to the third operation acting on the input device, the second device executes a control instruction corresponding to the third operation, and the third operation is used to input the control instruction on the second device.

2. The method according to claim 1, characterized in that In response to the first operation, the first device establishing a connection with a second device located in the first direction of the first device includes: In response to the first operation, the first device finds a second device that meets a connection condition in the first direction; The first device establishes a connection with the second device.

3. The method according to claim 2, characterized in that Before the first device detects a first operation acting on the input device, the method further includes: The first device performs device discovery; The first device stores, in the device table, a device identifier of the device that meets the connection condition and a direction of the device that meets the connection condition relative to the first device; In response to the first operation, the first device searches for a second device that meets a connection condition in the first direction, including: In response to the first operation, the first device searches the device table to find a second device that meets the connection condition in the first direction.

4. The method according to claim 3, characterized in that The method further comprises: The first device detects a change in orientation of the second device relative to the first device; The first device updates the direction of the second device relative to the first device in the device table.

5. The method according to any one of claims 2 to 4, characterized in that A third device is also included in the first direction of the first device, and before the first device establishes a connection with the second device, the method further includes: The first device detects a fourth operation of the user, where the fourth operation is used to select the second device.

6. The method according to any one of claims 1 to 5, characterized in that Before the first device detects a first operation acting on the input device, the method further includes: The first device displays a first interface, wherein the first interface includes an automatic connection option; In response to an operation acting on the automatic connection option, the automatic connection option is selected.

7. The method according to claim 6, characterized in that In the case where the automatic connection option is not selected, the method further includes: In response to an operation of opening a first application on the first device, the first device displays a second interface, where the second interface includes: a first icon for representing the first device, a second icon for representing the second device, and placement areas located on both sides of the first icon; In response to a fifth operation of dragging the second icon to a first placement area, the first device establishes a connection with the second device, and the first placement area is in a first direction of the first device.

8. The method according to claim 7, characterized in that In response to the fifth operation of dragging the second icon to the first placement area, establishing a connection between the first device and the second device includes: In response to the fifth operation, the first device determines whether there is a connected device in the first direction; If the connected device does not exist, the first device establishes a connection with the second device; If the connected device exists, the first device is prohibited from connecting to the second device, and the position of the second icon is restored to the position before dragging.

9. The method according to any one of claims 2 to 8, characterized in that The connection condition is that the login account of the second device is the same as that of the first device.

10. The method according to claim 1, characterized in that The method further comprises: In response to the second operation, the first device acquires a position where the cursor moves out of the first screen; The first device determines, based on the exit position, an initial position of the cursor entering the second screen; The first device sends the initial position to the second device to instruct the second device to display the cursor at the initial position on the second screen.

11. The method according to claim 10, characterized in that The first device determines, based on the exit position, an initial position of the cursor entering the second screen, including: The first device determines the initial position based on the exit position, a screen resolution of the first screen, and a screen resolution of the second screen.

12. The method according to claim 11, characterized in that The first device determines the initial position based on the exit position, the screen resolution of the first screen, and the screen resolution of the second screen, including: The first device determines a first conversion rate according to a screen resolution of the first screen and a screen resolution of the second screen; The first device determines the initial position based on the exit position and the first conversion rate.

13. The method according to claim 1, wherein The method further comprises: In response to the second operation, the first device acquires a first moving speed of the cursor on the first screen; The first device determines a second moving speed of the cursor on the second screen based on the first moving speed; The first device sends the second movement speed to the second device to instruct the second device to move the cursor on the second screen at the second movement speed.

14. The method according to claim 13, characterized in that The first device determines, based on the first movement speed, a second movement speed of the cursor on the second screen, including: The first device determines the second movement speed based on the first movement speed, a screen resolution of the first screen, and a screen resolution of the second screen.

15. The method according to claim 14, characterized in that The first device determines the second movement speed based on the first movement speed, the screen resolution of the first screen, and the screen resolution of the second screen, including: The first device determines a second conversion rate according to a screen resolution of the first screen and a screen resolution of the second screen; The first device determines the second moving speed based on the first moving speed and the second conversion rate.

16. The method according to claim 1, wherein After the first device controls the cursor to move out of the first screen and enter the second screen of the second device for display, the method further includes: The first device receives the third operation; intercepting, by the first device, an input event corresponding to the third operation; The first device sends the input event to the second device, wherein the second device determines the control instruction corresponding to the third operation according to the input event.

17. The method according to claim 16, characterized in that After, in response to the second operation, the first device controls the cursor to move out of the first screen and be displayed on the second screen of the second device, the method further includes: The first device receives a sixth operation applied to the input device, the sixth operation being used to drag the cursor displayed on the second screen toward the first device; In response to the sixth operation, the first device controls the cursor to move out of the second screen and be displayed on the first screen of the second device; The first device stops intercepting the input event corresponding to the operation acting on the input device to execute the control instruction corresponding to the operation of the input device.

18. The method according to claim 16, characterized in that After, in response to the second operation, the first device controls the cursor to move out of the first screen and be displayed on the second screen of the second device, the method further includes: The first device receives a seventh operation applied to the input device, the seventh operation being used to input a control instruction for forcibly reclaiming the cursor on the second device; In response to the seventh operation, the first device controls the cursor to be displayed on the first screen; The first device stops intercepting the input event corresponding to the operation acting on the input device to execute the control instruction corresponding to the operation of the input device.

19. The method according to claim 1, wherein The first direction is left or right, and after the first device establishes a connection with a second device located in the first direction of the first device, the method further includes: The first device detects that the second device moves from the first direction to a second direction opposite to the first direction; In response to an eighth operation, the first device controls the cursor to move out of the first screen and enter the second screen of the second device for display. The eighth operation is opposite to the direction of the cursor dragging by the second operation.

20. The method according to claim 1, wherein The first direction is left or right, and after the first device establishes a connection with a second device located in the first direction of the first device, the method further includes: The first device detects that the second device moves from the first direction to the middle; In response to a ninth operation, the first device controls the cursor to move out of the first screen and be displayed on the second screen of the second device. The ninth operation is in the same direction as the second operation of dragging the cursor.

21. The method according to claim 1, wherein The first direction is middle, and after the first device establishes a connection with a second device located in the first direction of the first device, the method further includes: The first device detects that the second device moves from the first direction to a second direction, where the second direction is left or right; In response to a tenth operation, the first device controls the cursor to move out of the first screen and enter the second screen of the second device for display, and the tenth operation is the same as the second direction.

22. The method according to claim 1, wherein The first direction is left or right, the first device is connected to a third device located in a second direction of the first device, and the second direction is opposite to the first direction; After the first device establishes a connection with a second device located in the first direction of the first device, the method further includes: The first device detects that the second device moves from the first direction to the second direction; In response to an eleventh operation, the first device controls the cursor to move out of the first screen and be displayed on the second screen of the second device. The eleventh operation is in the same direction as the second operation of dragging the cursor.

23. The method according to claim 1, wherein Before the first device detects a first operation acting on the input device, the method further includes: The first device discovers the second device; The first device obtains, by the device identifier of the second device, and a device angle of the second device relative to the first device; The first device determines the coordinate direction and the traversing direction of the second device according to the device angle; The first device records the device identifier, the device angle, the coordinate direction, and the crossing direction in a device table.

24. The method according to claim 23, wherein After the first device establishes a connection with a second device located in the first direction of the first device, the method further includes: The first device detects movement of the second device; When the first device detects that the second device is stationary, the first device continues to detect the device angle of the second device after it moves; If the device angle after the movement and the device angle before the movement do not belong to the same angle interval, the first device determines a coordinate direction after the movement and a crossing direction after the movement according to the device angle after the movement, wherein each coordinate direction corresponds to a preset angle interval; The first device updates the device table based on the coordinate direction after the movement and the crossing direction after the movement.

25. The method according to claim 24, characterized in that The coordinate direction includes: left, right, and center, the first direction is left or right, the first device determines the coordinate direction after movement and the crossing direction after movement according to the device angle after movement, and the method further includes: In the case where the coordinate direction after the movement is a second direction opposite to the first direction, if there is a third device connected in the second direction, the crossing direction after the movement is the same as the crossing direction before the movement; In the case where the coordinate direction after the movement is a second direction opposite to the first direction, if there is no third device connected in the second direction, the crossing direction after the movement is opposite to the crossing direction before the movement; or In the case where the coordinate direction after the movement is in the middle, the crossing direction after the movement is the default crossing direction or Same direction as before the move.

26. The method according to any one of claims 1 to 25, characterized in that The cursor is displayed as a first shape on the first screen, and is displayed as a second shape on the second screen, and the first shape and the second shape are different.

27. The method according to any one of claims 1 to 26, characterized in that The first device is a Windows system, and the second device is an Android system.

28. A terminal device, characterized in that: include: A processor, the processor is coupled to a memory, the memory is used to store a computer program, when the processor calls the computer program, the terminal device executes the method according to any one of claims 1 to 27.

29. A computer-readable storage medium, characterized in that Used to store a computer program, the computer program comprising instructions for implementing the method according to any one of claims 1 to 27.

30. A computer program product, characterized in that The computer program product includes computer program code, and when the computer program code is run on a computer, the computer is enabled to implement the method according to any one of claims 1 to 27.

Images