Keyboard and mouse crossing method, communication system and computer readable storage medium

Abstract

The invention provides a keyboard and mouse crossing method, a communication system and a computer readable storage medium. Comprising the steps that under the condition that a cursor of an input device is displayed on a screen of a first device, in response to a first operation, the first device controls the cursor to move out of the screen of the first device and enter a screen of a second device to be displayed, and the second device is located in the first direction of the first device and is connected with the first device; in response to the second operation, the second device controls the cursor to move out of the screen of the second device and enter a screen of a third device to be displayed, and the third device is located in the second direction of the second device and is connected with the second device. And the input device can pass through different devices, so that tedious operation is avoided, and the user experience is improved.

Description

Keyboard and mouse traversal method, communication system, and computer-readable storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 12, 2023, with application number 202310862162.5 and application name “Keyboard and mouse crossing method, communication system and computer-readable storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal control, and in particular to a keyboard and mouse traversal method, a communication system, and a computer-readable storage medium. Background Art

[0003] With the development of terminal technology, the types and number of terminal devices are increasing. Currently, a single terminal device is no longer sufficient for users, and users are using multiple devices to meet their work and study needs. However, when multiple terminals have tasks requiring user processing, users need to switch back and forth between different terminals, which is cumbersome and detrimental to user experience.

[0004] Summary of the Invention

[0005] The present application provides a keyboard and mouse traversal method, a communication system, and a computer-readable storage medium, which can operate different terminals through the keyboard and mouse to achieve seamless access between different terminal devices.

[0006] To achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, a keyboard and mouse method is provided, the method comprising:

[0008] In a case where a cursor of an input device is displayed on a screen of a first device, the first device detects a first operation applied to the input device, the first operation being used to move the cursor displayed on the screen of the first device in a first direction, wherein the input device is used to input a control instruction;

[0009] In response to the first operation, the first device controls the cursor to move out of the screen of the first device and to be displayed on the screen of a second device, wherein the second device is located in the first direction of the first device and is connected to the first device;

[0010] The second device detects a second operation applied to the input device, the second operation being used to move a cursor displayed on a screen of the second device in a second direction;

[0011] In response to the second operation, the second device controls the cursor to move out of the screen of the second device and to be displayed on the screen of a third device, wherein the third device is located in the second direction of the second device and is connected to the second device.

[0012] In the embodiment of the present application, the input device may include a mouse, a keyboard, a handwriting tablet, etc. For example, when the input device is a mouse, the cursor of the input device is a mouse pointer (or mouse cursor).

[0013] Exemplarily, the first device and the third device are devices running an Android system (Android devices), and the second device is a device running a Windows system (PC). The first device and the third device are respectively connected to the second device for communication.

[0014] For example, in some application scenarios, a PC can be the second device, Android device I can be the first device, and Android device II can be the third device; or a PC can be the second device, Android device II can be the first device, and Android device I can be the third device. In this application scenario, the second device is communicatively connected to the first device and the third device respectively.

[0015] The keyboard and mouse crossing method provided in this application allows input devices to cross between different devices. A set of input devices can be used to control different devices. There is no need to switch the operation mode back and forth between different terminal devices, which avoids cumbersome operations and is conducive to improving user experience.

[0016] In one implementation, the first direction and the second direction may include leftward or rightward. In another implementation, the first direction and the second direction may also include upward or downward.

[0017] In the embodiment of the present application, the first operation and the second operation may be operations of dragging or sliding the mouse.

[0018] In an implementation of the first aspect, in response to the first operation, the first device controlling the cursor to move out of the screen of the first device and to be displayed on the screen of the second device includes:

[0019] In response to the first operation, the first device finds a second device that establishes a connection with the first device in the first direction;

[0020] The first device controls the cursor to move out of the screen of the first device and to be displayed on the screen of the second device.

[0021] In this implementation, if the first device does not find the second device that has established a connection with the first device in the first direction, the keyboard and mouse traversal is not performed.

[0022] In the embodiment of the present application, the direction of movement of the cursor (the first direction) is the traversal direction, which is equivalent to determining whether there is a traversable device in the traversal direction. By performing the above-mentioned pre-judgment step, the probability of keyboard and mouse traversal failure can be effectively reduced.

[0023] In an implementation of the first aspect, the method further includes:

[0024] In response to the first operation, the first device determines whether the input device is connected to the first device;

[0025] If the input device is connected to the first device, the first device updates the crossing state of the first device to crossing;

[0026] The second device updates the crossing state of the second device to crossing.

[0027] In this implementation, an input device is connected to a first device. The first operation is to move the cursor on the first device's screen to the second device's screen. This is equivalent to operating the first device's input device, causing the first device's input device to pass through to the second device. By updating the pass-through status between the first and second devices, confusion in the pass-through status can be effectively reduced, thereby ensuring the reliability of keyboard and mouse pass-through.

[0028] In an implementation of the first aspect, after the first device determines, in response to the first operation, whether the input device is connected to the first device, the method further includes:

[0029] If the input device is connected to the first device, the first device detects a third operation, where the third operation is used to input the control instruction;

[0030] The first device intercepts a first input event corresponding to the third operation;

[0031] The first device sends the first input event to the second device;

[0032] After receiving the first input event, the second device executes the control instruction corresponding to the first input event.

[0033] In one implementation, when the first device is an Android device, the keyboard and mouse service module in the first device can call a preset interface of the input framework to intercept the first input event corresponding to the third operation.

[0034] In the embodiment of the present application, when a keyboard and mouse operation on a first device is passed through to a second device, the first device intercepts the input event corresponding to the keyboard and mouse operation and sends the input event to the second device. The second device then executes the control instruction corresponding to the operation on the input device. This allows different devices to share the input device, eliminating the need to switch between different operation modes between devices, reducing cumbersome operations, and improving the user experience. In addition, by intercepting the input event, the first device is prevented from responding to the keyboard and mouse operation, thereby ensuring the reliability of keyboard and mouse pass-through.

[0035] In an implementation of the first aspect, in response to the second operation, the second device controlling the cursor to move out of the screen of the second device and to be displayed on the screen of the third device includes:

[0036] In response to the second operation, the second device searches for a third device that has established a connection with the second device in the second direction;

[0037] The second device controls the cursor to move out of the screen of the second device and to be displayed on the screen of the third device.

[0038] In this implementation, if the second device does not find a third device that has established a connection with the second device in the second direction, keyboard and mouse traversal is not performed.

[0039] In the embodiment of the present application, the direction of movement of the cursor (the second direction) is the traversal direction, which is equivalent to determining whether there is a traversable device in the traversal direction. By performing the above-mentioned pre-judgment step, the probability of keyboard and mouse traversal failure can be effectively reduced.

[0040] In an implementation of the first aspect, after the first device determines, in response to the first operation, whether the input device is connected to the first device, the method further includes:

[0041] If the input device is connected to the first device, in response to a fourth operation, the second device determines whether the first device is the same as the third device, the fourth operation being used to move a cursor displayed on a screen of the second device in the second direction;

[0042] If the first device is the same as the third device, the first device updates the crossing status of the first device to not crossed;

[0043] The second device updates the crossing status of the second device to not crossing.

[0044] Exemplarily, the first device is the same as the third device, and the corresponding application scenario is that the keyboard and mouse of the first device are passed to the second device, and then passed back to the first device from the second device.

[0045] In the embodiment of the present application, by timely updating the crossing status of the first device and the second device, the occurrence of crossing confusion can be effectively reduced, which is conducive to improving the reliability of keyboard and mouse crossing.

[0046] In an implementation of the first aspect, after, in response to the fourth operation, the second device determines whether the first device is the same as the third device, the method further includes:

[0047] If the first device is the same as the third device, the first device stops intercepting the first input event corresponding to the third operation;

[0048] The first device executes the control instruction corresponding to the first input event.

[0049] In the embodiments of the present application, when a keyboard and mouse operation on a first device is passed through to a second device and then back to the first device, the first device stops intercepting input events corresponding to the keyboard and mouse operation and executes the control instructions corresponding to the input device operation. This allows for flexible switching of input devices between different devices, avoids cumbersome operations, and improves the user experience. Furthermore, by ceasing to intercept input events, the first device ensures timely response to keyboard and mouse operations, thereby ensuring the reliability of keyboard and mouse pass-through.

[0050] In an implementation of the first aspect, after, in response to the fourth operation, the second device determines whether the first device is the same as the third device, the method further includes:

[0051] If the first device is the same as the third device, the first device controls the cursor to be displayed at a first preset position on the screen of the first device.

[0052] In an embodiment of the present application, when the keyboard and mouse of the first device are transferred to the second device and then transferred back to the first device, the cursor is restored to the first preset position on the screen of the first device, which helps the user quickly find the position of the transferred cursor, thereby improving the user experience.

[0053] In an implementation of the first aspect, after, in response to the fourth operation, the second device determines whether the first device is the same as the third device, the method further includes:

[0054] If the first device is different from the third device, the second device sends a first instruction to the first device;

[0055] After receiving the first instruction, the first device controls the cursor to move out of the screen of the second device and to be displayed on the screen of the third device.

[0056] In the embodiment of the present application, the first instruction may be referred to as a transfer instruction, which is used to inform the source end that a receiving end of a keyboard and mouse event has changed.

[0057] In the embodiment of the present application, the first instruction is used to inform the first device that the receiving end has been transferred, so that the first device responds in time and controls the cursor to pass to the updated receiving end. This method is conducive to improving the efficiency and reliability of keyboard and mouse passing.

[0058] In an implementation of the first aspect, after receiving the first instruction, the first device controls the cursor to move out of the screen of the second device and to be displayed on the screen of the third device, including:

[0059] After receiving the first instruction, the first device establishes a connection with the third device;

[0060] After the first device establishes a connection with the third device, the first device controls the cursor to move out of the screen of the second device and to be displayed on the screen of the third device.

[0061] In an embodiment of the present application, when the receiving end is transferred, the first device establishes a connection with the updated receiving end (the third device), thereby avoiding the first device communicating with the second device first and then forwarding it to the third device by the second device, effectively simplifying the communication process and helping to improve the keyboard and mouse response speed.

[0062] In an implementation of the first aspect, after the first device establishes a connection with the third device, the method further includes:

[0063] The second device updates the crossing status of the second device to not crossing;

[0064] The third device updates the crossing state of the third device to crossing.

[0065] In this implementation, by updating the crossing status of the second device and the third device, the occurrence of confusion in the crossing status can be effectively reduced, thereby ensuring the reliability of keyboard and mouse crossing.

[0066] In an implementation of the first aspect, after the first device establishes a connection with the third device, the method further includes:

[0067] If the first device is disconnected from the third device, the first device controls the cursor to be displayed at a first preset position on the screen of the first device;

[0068] The first device maintains a communication connection with the second device.

[0069] Optionally, if the first device is disconnected from the third device, the first device restores the keyboard and mouse positions.

[0070] In one implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the Android device can be restored based on the positions of the keyboard and mouse on the screen of the Android device that are transferred from the PC back to the device.

[0071] In another implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the Android device can be restored according to the keyboard and mouse positions that are crossed from the Android device to the PC screen.

[0072] In another implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the Android device can be restored to a preset position. For example, the preset position is the center of the screen of the Android device.

[0073] In an embodiment of the present application, when the keyboard and mouse of the first device pass through to the third device and the first device is disconnected from the third device, the cursor is restored to the first preset position on the screen of the first device, which helps the user quickly find the position of the passed cursor, thereby improving the user experience.

[0074] In an implementation of the first aspect, after the first device establishes a connection with the third device, the method further includes:

[0075] If the first device is disconnected from the third device, the first device stops intercepting the input event corresponding to the third operation;

[0076] The first device executes the control instruction corresponding to the input event.

[0077] In the embodiments of the present application, if a keyboard or mouse operation on a first device is passed through to a third device and the first and third devices are disconnected, the first device stops intercepting input events corresponding to the keyboard or mouse operation and executes the control instructions corresponding to the input device operation. This allows for flexible switching of input devices between different devices, avoids cumbersome operations, and improves the user experience. Furthermore, by ceasing to intercept input events, the first device ensures timely response to keyboard or mouse operations, thereby ensuring the reliability of keyboard or mouse pass-through.

[0078] In an implementation of the first aspect, after the first device determines, in response to the first operation, whether the input device is connected to the first device, the method further includes:

[0079] If the input device is not connected to the first device, the first device updates the crossing state of the first device to not crossed.

[0080] For example, the corresponding application scenario may be that the cursor of the keyboard and mouse of another device is displayed on the screen of the first device and passes from the first device to the second device.

[0081] In the embodiment of the present application, by timely updating the traversal status of the first device, the occurrence of traversal confusion can be effectively reduced, which is conducive to improving the reliability of keyboard and mouse traversal.

[0082] In an implementation of the first aspect, after the first device determines, in response to the first operation, whether the input device is connected to the first device, the method further includes:

[0083] If the input device is not connected to the first device, the first device sends a second instruction to the second device;

[0084] After receiving the second instruction, the second device determines whether the input device is connected to the second device;

[0085] If the input device is connected to the second device, the second device controls the cursor to move out of the screen of the first device and display it at a second preset position on the screen of the second device.

[0086] In the embodiment of the present application, the second instruction may be referred to as a return instruction. The return instruction is used to inform the second device that the mouse pointer passes back to the second device.

[0087] For example, the corresponding application scenario may be that the cursor of the keyboard and mouse of the second device is displayed on the screen of the first device, and then passed back from the first device to the second device.

[0088] In an embodiment of the present application, the cursor is restored to a second preset position on the screen of the second device, which helps the user quickly find the position of the crossing cursor, thereby improving the user experience.

[0089] In an implementation of the first aspect, after determining whether the input device is connected to the second device, the method further includes:

[0090] If the input device is connected to the second device, the second device detects a fifth operation, where the fifth operation is used to input the control instruction;

[0091] The second device stops intercepting the second input event corresponding to the fifth operation;

[0092] The second device executes the control instruction corresponding to the second input event.

[0093] For example, the corresponding application scenario may be that the cursor of the keyboard and mouse of the second device is displayed on the screen of the first device, and then passed back from the first device to the second device.

[0094] In this embodiment of the present application, the second device stops intercepting input events corresponding to keyboard and mouse operations and executes control instructions corresponding to the input device operations. This allows for flexible switching of input devices between different devices, avoids cumbersome operations, and improves the user experience. Furthermore, by stopping interception of input events, the second device is guaranteed to respond to keyboard and mouse operations in a timely manner, thereby ensuring the reliability of keyboard and mouse cross-connection.

[0095] In an implementation of the first aspect, after determining whether the input device is connected to the second device, the method further includes:

[0096] If the input device is connected to the second device, the second device updates the crossing state of the second device to not crossing.

[0097] For example, the corresponding application scenario may be that the cursor of the keyboard and mouse of the second device is displayed on the screen of the first device, and then passed back from the first device to the second device.

[0098] In the embodiment of the present application, by timely updating the traversal status of the second device, the occurrence of traversal confusion can be effectively reduced, which is conducive to improving the reliability of keyboard and mouse traversal.

[0099] In an implementation of the first aspect, after determining whether the input device is connected to the second device, the method further includes:

[0100] If the input device is connected to the second device, in response to the sixth operation, the second device determines a first traversal position on the screen of the second device, and the sixth operation is used to move a cursor displayed on the screen of the second device in the second direction;

[0101] The second device sends the first crossing location to the third device;

[0102] The third device determines a second crossing position on the screen of the third device according to the first crossing position;

[0103] The second device controls the cursor to move from the first crossing position out of the screen of the second device;

[0104] The third device controls the cursor to enter the screen of the third device from the second traversal position.

[0105] For example, a corresponding application scenario may be that the cursor of the keyboard and mouse of the second device is displayed on the screen of the first device, and then passed back from the first device to the second device.

[0106] For example, if the mouse pointer's crossing position on the PC screen is (x1, y1) and the crossing direction is left crossing or right crossing, the PC's conversion module can be based on s y Change the transformation of y1, that is, y1×s y The value of the horizontal axis is a preset value, which is the minimum or maximum value of the screen side length on the X axis. R 1y is the side length of the y-axis of the Android device screen, R 2y The length of the y-axis of the PC screen.

[0107] If the mouse's crossing position on the PC screen is (x1, y1) and the crossing direction is up or down, the PC's conversion module can be based on s x Change the transformation of x1, that is, x1×s x The value of the vertical coordinate is a preset value, which is the minimum or maximum value of the screen side length on the X axis. R 1x is the length of the x-axis of the Android device screen, R 2x The length of the x-axis of the PC screen.

[0108] Through the above method, when the resolutions of the second device and the third device are different, the crossing position can be accurately determined, and the reliability of keyboard and mouse crossing is improved by improving the orderliness of the crossing.

[0109] In an implementation of the first aspect, after responding to the sixth operation, the method further includes:

[0110] The second device updates the crossing state of the second device to crossing.

[0111] By timely updating the traversal status of the second device, the occurrence of traversal confusion can be effectively reduced, which is conducive to improving the reliability of keyboard and mouse traversal.

[0112] In an implementation of the first aspect, after responding to the sixth operation, the method further includes:

[0113] The second device detects a seventh operation, where the seventh operation is used to input the control instruction;

[0114] The second device intercepts a third input event corresponding to the seventh operation;

[0115] The second device sends the third input event to the third device;

[0116] After receiving the third input event, the third device executes a control instruction corresponding to the third input event.

[0117] Exemplarily, the corresponding application scenario may be that the keyboard and mouse of the second device are transferred to the third device.

[0118] In one implementation, if the second device is a Windows-based device, the second device can intercept the third input event corresponding to the seventh operation through an interception event. The interception event is used to intercept keyboard and mouse events. The interception event indication information is used to instruct the input framework to set keyboard hooks, mouse hooks, and shortcut hooks. A hook is a platform within the Windows message processing mechanism where applications can set subroutines to monitor specific messages from a specified window. The monitored window can be created by another process. When a message arrives, it is processed before the target window's handler function. The hook mechanism allows applications to intercept and process window messages or specific events. A hook is essentially a message processing program segment that is hooked into the system through a system call. Whenever a specific message is sent, before it reaches the target window, the hook program captures the message, effectively giving the hook function control. The hook function can then process (modify) the message, pass it on without processing, or forcibly terminate message delivery.

[0119] The keyboard hook is used to obtain keyboard input operations, for example, the keyboard hook is used to obtain the user's keyboard input "memo information" operation. The mouse hook is used to obtain the user's mouse control operations, for example, the mouse hook is used to obtain the user's mouse click or drag operation. The shortcut key hook is used to obtain the user's shortcut key operation, for example, the shortcut key hook is used to obtain the user's Ctrl+C, Ctrl+V, ATL+TAB, or Ctrl+Alt+DELETE operation.

[0120] By intercepting events, the mouse passes from the PC to the Android device. The Windows system input framework intercepts the monitored keyboard and mouse events, making the business on the PC unresponsive to keyboard and mouse events, so that the PC does not affect the mouse events.

[0121] The second device intercepts input events corresponding to keyboard and mouse operations and sends them to the third device, which then executes the control instructions corresponding to the operations on the input device. This allows different devices to share input devices, eliminating the need to switch between different operation modes, reducing cumbersome operations, and improving the user experience. Furthermore, by intercepting input events, the second device is prevented from responding to keyboard and mouse operations, thus ensuring the reliability of keyboard and mouse cross-connection.

[0122] In an implementation of the first aspect, after determining whether the input device is connected to the second device, the method further includes:

[0123] If the input device is not connected to the second device, the second device determines a fourth device connected to the input device;

[0124] The second device sends a third instruction to the fourth device;

[0125] After receiving the third instruction, the fourth device controls the cursor to move out of the screen of the first device and to be displayed on the screen of the second device.

[0126] In the embodiment of the present application, the third instruction may be referred to as a switching notification, which is used to inform the fourth device that the keyboard and mouse have been switched from the first device to the second device.

[0127] In one implementation, the second device sending the third instruction to the fourth device includes: the second device establishing a communication connection with the fourth device; after the second device establishes the communication connection with the fourth device, the second device sending the third instruction to the fourth device.

[0128] In some application scenarios, the fourth device can be the third device. For example, the corresponding application scenario is that the keyboard and mouse of the third device are passed to the first device, from the first device to the second device, and then from the second device back to the third device.

[0129] In this embodiment of the present application, the second device can determine the device (i.e., the source) to which the input device belongs, and thereby send a third instruction to the source to instruct the source to transfer the relevant data of the keyboard and mouse pass-through to the second device. This method can realize the sharing of input devices by different devices, eliminating the need to switch different operation modes between different devices, avoiding cumbersome operations, and helping to ensure the reliability of keyboard and mouse pass-through and improve the user experience.

[0130] In an implementation of the first aspect, in response to the first operation, the first device controlling the cursor to move out of the screen of the first device and to be displayed on the screen of the second device includes:

[0131] In response to the first operation, the first device determines whether a cursor on a screen of the first device reaches an edge of the screen;

[0132] If the cursor on the screen of the first device reaches an edge of the screen, the first device controls the cursor to move out of the screen of the first device and to be displayed on the screen of the second device.

[0133] In one implementation, determining whether the cursor has reached the edge of the screen includes: obtaining the current position of the cursor; obtaining the displacement of the cursor; calculating the position of the cursor after movement based on the current position of the cursor and the displacement of the cursor; determining whether the position of the cursor after movement exceeds the screen; if the position of the cursor after movement exceeds the screen, determining that the cursor has reached the edge of the screen.

[0134] It should be noted that if the first device determines whether the cursor has reached the edge of the screen, the above determination process is performed by the first device. If the second device determines whether the cursor has reached the edge of the screen, the above determination process is performed by the second device.

[0135] In a second aspect, a communication system is provided, comprising a first device, a second device, and a third device;

[0136] The second device is connected to the first device and the third device respectively;

[0137] The first device, the second device, and the third device in the communication system are configured to execute the method as described in any one of the first aspects.

[0138] According to a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of the first aspects is implemented.

[0139] According to a fourth aspect, a computer program product is provided, wherein the computer program product includes a computer program code, and when the computer program code is executed on a computer, the computer implements the method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0140] FIG1 is a schematic diagram of a communication system provided in an embodiment of the present application;

[0141] FIG2 is a schematic diagram of a software architecture of a Windows system provided in an embodiment of the present application;

[0142] FIG3 is a schematic diagram of a software architecture of an Android system provided in an embodiment of the present application;

[0143] FIG4 is a schematic diagram of an interaction for establishing a keyboard-mouse connection according to an embodiment of the present application;

[0144] FIG5 is a schematic diagram of an interface of a computer manager provided in an embodiment of the present application;

[0145] FIG6 is a schematic diagram of a position provided in an embodiment of the present application;

[0146] FIG7 is a schematic diagram of a position provided by another embodiment of the present application;

[0147] FIG8 is a schematic diagram of a keyboard and mouse traversal method provided in an embodiment of the present application;

[0148] FIG9 is a schematic diagram of a keyboard and mouse traversal scenario provided by an embodiment of the present application;

[0149] FIG10 is a schematic diagram of an interaction scenario provided in an embodiment of the present application;

[0150] FIG11 is a schematic diagram of an implementation principle of a traversal method provided in an embodiment of the present application;

[0151] FIG12 is a schematic diagram of an interaction flow of a keyboard and mouse traversal method provided in an embodiment of the present application;

[0152] FIG13 is a schematic diagram of a screen crossing position provided by an embodiment of the present application;

[0153] FIG14 is a schematic diagram of another keyboard and mouse traversal scenario provided by an embodiment of the present application;

[0154] FIG15 is a schematic diagram of an implementation principle of a traversal method provided in an embodiment of the present application;

[0155] FIG16 is a schematic diagram of an interaction flow of a keyboard and mouse traversal method provided in an embodiment of the present application;

[0156] FIG17 is a schematic diagram of a keyboard and mouse traversal scenario provided by an embodiment of the present application;

[0157] FIG18 is a schematic diagram of a keyboard and mouse traversal scenario provided by an embodiment of the present application;

[0158] FIG19 is a schematic diagram of an interaction flow of a keyboard and mouse traversal method provided in an embodiment of the present application;

[0159] Figure 20 is a schematic diagram of the interaction process of a keyboard and mouse traversal method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0160] In the following description, specific details such as specific system structures and technologies are provided for illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details.

[0161] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0162] It should also be understood that in the embodiments of this application, "one or more" refers to one, two, or more than two; "and / or" describes the relationship between associated objects, indicating that three relationships can exist; for example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0163] In addition, in the description of this application specification and the appended claims, the terms "first", "second", "third", "fourth", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0164] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0165] With the development of terminal technology, the types and number of terminal devices are increasing. Currently, a single terminal device is no longer sufficient for users, and users are using multiple devices to meet their work and study needs. However, when multiple terminals have tasks requiring user processing, users need to switch back and forth between different terminals, which is cumbersome and detrimental to user experience.

[0166] The embodiment of the present application provides a keyboard and mouse traversal method, which can operate different terminal devices through a set of input devices to achieve seamless access between different terminal devices.

[0167] The methods of the embodiments of the present application can be applied to terminal devices with screens and access to input devices. For example, the terminal devices in the embodiments of the present application can be personal computers (PCs) (such as desktop computers, laptops, small laptops, or ultrabooks), tablet computers (referred to as tablets), and mobile phones. The embodiments of the present application do not specifically limit the specific form of the terminal devices. For ease of explanation, the following embodiments are introduced using PCs (or computers), tablets, and mobile phones as examples.

[0168] In the embodiments of the present application, the input device may be a keyboard, a mouse, a tablet (which may include a stylus), or other device used to input information into a terminal device. The embodiments of the present application do not specifically limit the specific form of the input device. For ease of explanation, the following embodiments refer to the keyboard and mouse as simply keyboard and mouse, and the keyboard and mouse are used as an example for the description of the input device.

[0169] In the embodiment of the present application, keyboard and mouse crossover can be implemented between two or more terminal devices. The following is an example of three terminal devices.

[0170] Referring to Figure 1, which is a schematic diagram of a communication system according to an embodiment of the present application, the communication system includes a computer 11, a mobile phone 12, and a tablet 13. The computer 11 is connected to the mobile phone 12 and the tablet 13 in communication.

[0171] In the communication system shown in Figure 1, keyboard and mouse can be passed between the computer 11 and the mobile phone 12, between the computer 11 and the tablet 13, and between the mobile phone 12 and the tablet 13 via the computer 11. In other words, the computer 11, the mobile phone 12, and the tablet 13 can share a mouse and keyboard.

[0172] It is understandable that the computer 11 can also achieve keyboard and mouse crossover only with the mobile phone 12 or the tablet 13. In other words, the computer 11 and the mobile phone 12 share the mouse and keyboard, or the computer 11 and the tablet 13 share the mouse and keyboard.

[0173] In the embodiment of the present application, the keyboard and mouse of the computer 11 may be a keyboard and mouse connected to the computer 11 , and the keyboard and mouse of the mobile phone 12 or tablet 13 may be a keyboard and mouse external to the mobile phone 12 or tablet 13 .

[0174] The left or right side of the computer 11 can achieve keyboard and mouse pass-through with a single device. For example, the computer 11 can support mouse pass-through from the left side of the computer 11 screen to the tablet 13, or from the right side of the computer 11 screen to the mobile phone 12. The computer 11 can achieve keyboard and mouse pass-through with one device (mobile phone 12 or tablet 13) at a time. In other words, if a device is currently in the pass-through state, keyboard and mouse pass-through will no longer be possible with other devices.

[0175] In the embodiment of the present application, the software system of the terminal device involved in keyboard and mouse crossover can be a Windows system or an Android system. For example, the operating system of the computer 11 shown in Figure 1 is a Windows system, and the operating system of the mobile phone 12 and the tablet 13 is an Android system. Among them, the device deploying the Android system can be called an Android device.

[0176] In order to better understand the embodiments of the present application, the software architecture of the Windows system and Android system of the embodiments of the present application are introduced below.

[0177] For example, see Figure 2, which is a schematic diagram of a software architecture of a Windows system provided by an embodiment of the present application. As shown in Figure 2, the software architecture of the Windows system may include a driver layer, a basic framework layer, and an application layer.

[0178] The driver layer includes a virtual driver module and an input driver module. The virtual driver module is used to set up virtual drivers for the input devices of the Android device. The input driver module is used to provide drivers for human interface devices (HID) to drive input devices connected to the PC, such as keyboards and mice.

[0179] The basic framework layer includes an input framework module, which reports input events to the various service modules in the application layer, enabling them to implement their corresponding functions. For example, the input framework module reports keyboard and mouse events to the keyboard and mouse service module in the application layer, which then processes the events and implements keyboard and mouse traversal.

[0180] The application layer includes a communication module (such as Magic Link) and a keyboard and mouse service module. The communication module is used to provide a connection channel and data transmission capability between devices. The keyboard and mouse service module is used to provide a keyboard and mouse crossing function. Among them, the keyboard and mouse service module includes a connection scheduling submodule, a keyboard and mouse management submodule and a virtual driver loading submodule. The connection scheduling submodule is used to provide a connection channel and data transmission capability with the communication module. The keyboard and mouse management submodule is used to call the connection scheduling submodule and the virtual driver loading submodule, and provide the ability to process keyboard and mouse events. The virtual driver loading submodule is used to inject the keyboard and mouse events processed by the keyboard and mouse management submodule into the virtual driver module in the driver layer, so as to report the keyboard and mouse events to the input framework module in the system base layer through the virtual driver module.

[0181] It should be noted that the communication module is a module for communication between different devices. Data obtained by other modules must first be transmitted to the communication module before being transmitted to other devices. In subsequent embodiments, to simplify the steps, in addition to communication modules, when other modules communicate across devices, this can be understood as other modules first transmitting data to the communication module, which then transmits the data to the communication module of the other device, which then transmits the data to the module that receives the data.

[0182] It should be noted that the data interaction between the keyboard and mouse service module and the communication module needs to go through the connection scheduling submodule. For example, the data packet of the keyboard and mouse event sent by other devices is first transmitted to the communication module, and then sent by the communication module to the connection scheduling submodule, and then sent by the connection scheduling submodule to the keyboard and mouse management submodule. In subsequent embodiments, in order to simplify the steps, in the case where other submodules other than the connection scheduling submodule in the keyboard and mouse service module communicate with the communication module, it can be understood that other submodules first transmit data to the connection scheduling submodule, and then the connection scheduling submodule transmits it to the communication module, or the communication module first transmits data to the connection scheduling submodule, and then the connection scheduling submodule transmits it to other submodules.

[0183] For example, see Figure 3, which is a schematic diagram of a software architecture of an Android system provided by an embodiment of the present application. As shown in Figure 3, the software architecture of the Android system may include a driver layer, a basic framework layer, and an application layer.

[0184] The driver layer includes a virtual input driver module (Uinput Driver), which is used to drive the input device of the Android device.

[0185] The basic framework layer includes an input framework module, which reports input events to the various service modules in the application layer, enabling them to implement their corresponding functions. For example, the input framework module reports keyboard and mouse events to the keyboard and mouse service module in the application layer, which then processes the events and implements keyboard and mouse traversal.

[0186] The application layer includes a communication module (such as Magic Link) and a keyboard and mouse service module. The communication module is used to provide a connection channel and data transmission capability between devices. The keyboard and mouse service module is used to provide a keyboard and mouse crossing function. Among them, the keyboard and mouse service module includes a connection scheduling submodule and a keyboard and mouse management submodule. The connection scheduling submodule is used to provide a connection channel and data transmission capability with the communication module. The keyboard and mouse management submodule is used to call the connection scheduling submodule, provide the ability to process keyboard and mouse events, and inject the processed keyboard and mouse events into the virtual input driver module in the driver layer, so as to report the keyboard and mouse events to the input framework module in the system base layer through the virtual input driver module.

[0187] It should be noted that the communication module is a module for communication between different devices. Data obtained by other modules must first be transmitted to the communication module before being transmitted to other devices. In subsequent embodiments, to simplify the steps, in addition to communication modules, when other modules communicate across devices, this can be understood as other modules first transmitting data to the communication module, which then transmits the data to the communication module of the other device, which then transmits the data to the module that receives the data.

[0188] It should be noted that the data interaction between the keyboard and mouse service module and the communication module needs to go through the connection scheduling submodule. For example, the data packet of the keyboard and mouse event sent by other devices is first transmitted to the communication module, and then sent by the communication module to the connection scheduling submodule, and then sent by the connection scheduling submodule to the keyboard and mouse management submodule. In subsequent embodiments, in order to simplify the steps, in the case where other submodules other than the connection scheduling submodule in the keyboard and mouse service module communicate with the communication module, it can be understood that other submodules first transmit data to the connection scheduling submodule, and then the connection scheduling submodule transmits it to the communication module, or the communication module first transmits data to the connection scheduling submodule, and then the connection scheduling submodule transmits it to other submodules.

[0189] In order to better understand the embodiments of the present application, the embodiments of the present application are combined with the communication system shown in Figure 1 and the software architecture shown in Figures 2 and 3 to specifically introduce the method provided in the embodiments of the present application.

[0190] In an embodiment of the present application, the condition for achieving keyboard and mouse crossing between terminal devices may be that a communication connection has been established between the terminal devices. For example, the PC and the Android device can be connected via WiFi or Bluetooth communication, where WiFi may include a local area network or person-to-person (peer to pee, P2P); or they can be connected via a wired communication, such as via a data cable such as type-A, type-B or type-C. In other implementations, the PC and the Android device use the same account to log in. For example, when the computer 11, mobile phone 12 and tablet 13 use the same account to log in, the computer 11, mobile phone 12 and tablet 13 establish a communication connection.

[0191] For example, see Figure 4, which is a schematic diagram of the interaction of establishing a keyboard and mouse connection provided by an embodiment of the present application. As shown in Figure 4, the process of establishing a keyboard and mouse connection between a PC and any Android device may include the following steps:

[0192] S401, the keyboard and mouse management submodule of the PC is started.

[0193] The keyboard and mouse management submodule within the PC's keyboard and mouse service module can be automatically activated when the PC is turned on, or it can be manually activated by the user. For example, in the case of manual activation, the user can enable the keyboard and mouse pass-through feature through the settings interface on the PC. In response to the user enabling the keyboard and mouse pass-through feature, the PC's keyboard and mouse service module is activated, and accordingly, the keyboard and mouse management submodule within the PC's keyboard and mouse service module is activated.

[0194] S402: The keyboard and mouse management submodule of the PC sends instruction information a to the connection scheduling submodule of the PC.

[0195] The instruction information a is used to instruct the connection scheduling submodule of the PC to establish a communication connection with the Android device.

[0196] S403: After receiving the instruction information a, the connection scheduling submodule of the PC sends a request information b to the communication module of the PC.

[0197] The request information b is used to request the communication module of the PC to inform the detected online Android device.

[0198] The communication module of the PC can detect whether there is an online Android device. If there is an online Android device, based on the request information b, the communication module of the PC sends the device identity (ID) of the online Android device to the keyboard and mouse management submodule of the PC via the connection scheduling submodule of the PC. If there is no online Android device, the communication module of the PC can monitor in real time whether there is an online Android device, or detect whether there is an online Android device at a preset period.

[0199] The connection scheduling submodule of the PC sends a request message b to the communication module of the PC. It can also be understood that the connection scheduling submodule of the PC registers the device online and offline fence in the communication module of the PC. When the communication module of the PC detects that there is an online device, it returns the device ID to the keyboard and mouse management submodule of the PC through the connection scheduling submodule.

[0200] S404, the keyboard and mouse management submodule of the Android device is started.

[0201] The keyboard and mouse management submodule within the keyboard and mouse service module of an Android device can be automatically activated when the Android device is powered on, or it can be manually activated by the user. For example, in the case of manual activation, the user can enable the keyboard and mouse pass-through feature through the settings interface on the Android device. In response to the user enabling the keyboard and mouse pass-through feature, the keyboard and mouse service module of the Android device is activated, and accordingly, the keyboard and mouse management submodule within the keyboard and mouse service module of the Android device is activated.

[0202] S405 , after receiving the instruction information b, the communication module of the PC discovers the communication module of the Android device and forms a trust loop.

[0203] As mentioned above, keyboard and mouse traversal between a PC and an Android device requires an established communication connection between the PC and the Android device. This communication connection means that the PC's communication module and the Android device's communication module can discover each other and establish a trust loop.

[0204] It should be noted that the above-mentioned S402 is a step that will be executed after the keyboard and mouse management submodule of the PC is started. The embodiment of the present application does not limit the order in which the two steps S402 and S404 are executed. In other words, if before S402, step S404 has been executed, that is, the keyboard and mouse management submodule of the Android device has been started, then after executing S402, the communication module of the PC can detect that the Android device is online, and thus S405 can be executed. If before S402, step S404 has not been executed, that is, the keyboard and mouse management submodule of the Android device is not started, then after executing S402, the communication module of the PC detects in real time or at preset intervals whether there is an online Android device, until the keyboard and mouse management submodule of the Android device is started, and the communication module of the PC detects that there is an online Android device, then S405 is executed.

[0205] A trust ring indicates a communication link between different devices. For example, if a PC and an Android device are within the same Bluetooth range, connected to the same Wi-Fi network, connected via the same data cable, or logged into the same account, the PC's communication module and the Android device's communication module can sense and discover each other, forming a trust ring. The PC's communication module can then obtain the Android device's device ID.

[0206] The device ID can be at least one of numbers, letters, or symbols, which is not limited in this embodiment of the present application.

[0207] There may be one or more online Android devices, which is not limited in this embodiment of the present application. If there are multiple online Android devices, the communication module of the PC may send the device IDs of the multiple Android devices to the keyboard and mouse service module of the PC.

[0208] In one implementation, the keyboard and mouse service module of the PC may include a device management submodule. If the PC is communicating with an Android device for the first time, the device management submodule of the PC may authenticate the Android device based on the device ID. After the authentication is passed, the connection scheduling submodule of the PC notifies the communication module of the PC to execute S405 and store the device ID. During the next communication connection process, when the communication module of the PC detects the online Android device, the device ID of the Android device is reported to the device management submodule of the PC through the connection scheduling submodule of the PC. The device management submodule of the PC searches to see if the device ID has been stored. If the device ID has been stored, the connection scheduling submodule of the PC sends an instruction to the communication module of the PC to instruct the communication module of the PC to execute S405.

[0209] S406: After the trust ring is formed, the communication module of the Android device sends a notification c to the connection scheduling submodule of the Android device.

[0210] Notification c is used to instruct the connection scheduling submodule of the Android device to report to the keyboard and mouse management submodule of the Android device that the communication connection with the PC is successful.

[0211] S407 , after receiving notification c, the connection scheduling submodule of the Android device sends notification d to the keyboard and mouse management submodule of the Android device.

[0212] Notification d is used to indicate that the communication connection with the PC is successful.

[0213] S408: After the trust ring is formed, the communication module of the PC sends a notification e to the connection scheduling submodule of the PC.

[0214] Notification e is used to instruct the connection scheduling submodule of the PC to report to the keyboard and mouse management submodule of the PC that the communication connection with the Android device is successful.

[0215] S409: After receiving the notification e, the connection scheduling submodule of the PC sends a notification f to the keyboard and mouse management submodule of the PC.

[0216] Notification f is used to indicate that the communication connection with the Android device is successful.

[0217] It should be noted that steps S406 and S408 are not performed in any particular order and can be processed in parallel.

[0218] At this point, the keyboard and mouse connection between the PC and Android device is successful.

[0219] It is understandable that to achieve keyboard and mouse traversal between a PC and an Android device, it is necessary not only to establish a communication connection between the PC and the Android device (i.e., to form a trust loop between the PC's communication module and the Android device's communication module), but also to establish a keyboard and mouse connection between the PC and the Android device (i.e., an event connection between the PC's keyboard and mouse service module and the Android device's keyboard and mouse service module). In other words, Figure 4 shows the process of establishing a keyboard and mouse connection between a PC and an Android device. A successful keyboard and mouse connection between the PC and the Android device indicates a successful event connection between the PC's keyboard and mouse service module and the Android device's keyboard and mouse service module.

[0220] It should be noted that the embodiment of Figure 4 shows the process of establishing a keyboard and mouse connection with the PC as the center. In other words, the keyboard and mouse connection is initiated by the PC. In an application scenario centered on the PC, if keyboard and mouse crossing between the PC and multiple Android devices is to be achieved, it is necessary for the PC to establish a keyboard and mouse connection with each Android device respectively. In other embodiments, a keyboard and mouse connection can also be established with the Android device as the center, that is, the keyboard and mouse connection is initiated by the Android device. In an application scenario centered on the Android device, if keyboard and mouse crossing between the Android device and multiple terminal devices is to be achieved, it is necessary for the Android device to establish a keyboard and mouse connection with each terminal device respectively. For the sake of convenience, the application scenarios involved in the embodiments of the present application are all PC-centered scenarios.

[0221] There are two ways to establish a keyboard and mouse connection between a PC and an Android device. One possible implementation is to automatically establish a keyboard and mouse connection. In this way, after the keyboard and mouse management submodule of the PC is started in S401, the subsequent steps are automatically executed, and the PC can independently select a connectable device. Another possible implementation is to manually establish a keyboard and mouse connection. In this way, after the keyboard and mouse management submodule of the PC is started in S401, when the communication module of the PC detects an online Android device, the user can manually select a connectable device in the visual interface. After the user selects, the steps after S401 are executed.

[0222] The setting options for the keyboard and mouse connection mode may be located in different locations on different devices. For example, the setting options for the keyboard and mouse connection mode may be in the computer manager of the computer 11. The setting options for the keyboard and mouse connection mode may be in the control center or settings of the mobile phone 12 or tablet 13.

[0223] For example, see Figure 5, which is a schematic diagram of the interface of a computer manager provided in an embodiment of the present application. As shown in Figure 5, the computer manager interface may include options such as home page, multi-screen collaboration, smart interconnection, online skills, official services, smart audition, and system optimization. When the computer 11 detects that the user clicks on the smart interconnection option with the mouse, the keyboard and mouse sharing interface can be displayed. The keyboard and mouse sharing interface includes two options. Among them, the first option is "Touch the mouse pointer twice in a row on the side edge of the computer screen to establish a connection." The second option is "Automatically adjust the screen arrangement as the device placement changes."

[0224] It is understood that the first option is the option for automatically establishing a keyboard and mouse connection. When this option is selected, the computer 11 can automatically connect to devices that meet the keyboard and mouse pass-through conditions. When the first option (i.e., the option for automatically establishing a communication connection) is not selected, the computer 11 provides a method for manually establishing a keyboard and mouse connection to connect to devices that meet the keyboard and mouse pass-through conditions.

[0225] As an example of manually establishing a keyboard and mouse connection, as shown in Figure 5, the first option is unselected. In this case, the computer 11 provides a manual connection method. The keyboard and mouse sharing interface can display connected devices and online devices. In the keyboard and mouse sharing interface shown in Figure 5 (a), a mobile phone 12 is connected to the left side of the computer 11, and there are two online devices (devices with established trust loops), including a tablet 13 and a mobile phone 14. The user can select a device from these two online devices by clicking or dragging. As shown in Figure 5 (a), the user operates the mouse to control the mouse pointer to click on the icon of the tablet 13. In response to the operation of clicking the icon of the tablet 13, the computer 11 establishes a keyboard and mouse connection with the tablet 13 and displays the keyboard and mouse sharing interface shown in Figure 5 (b). As shown in Figure 5 (b), in the keyboard and mouse sharing interface, the tablet 13 is connected to the right side of the computer 11.

[0226] In the keyboard and mouse sharing interface as shown in (a) and (b) in Figure 5, a selection mark 501 can be displayed on the icon of the device connected to the computer 11. The user can disconnect the connection between the PC and the device by operating the mouse to control the mouse pointer to click the selection mark 501 or drag the icon of the connected device. For example, the user operates the mouse to click the selection mark 501 on the mobile phone on the left side of the computer 11 in the interface as shown in (b) in Figure 5. In response to the user operation, the computer 11 disconnects the keyboard and mouse connection with the mobile phone 12 and displays the keyboard and mouse sharing interface as shown in (c) in Figure 5. As shown in (c) in Figure 5, in the keyboard and mouse sharing interface, no device is connected to the left side of the computer 11, a tablet 13 is connected to the right side of the computer 11, and the mobile phone 12 is displayed in the area where the device can be connected.

[0227] When manually establishing a keyboard and mouse connection, the PC determines the device's traversal direction based on the position of the device selected by the user in the keyboard and mouse sharing interface. For example, as shown in FIG5(b), if mobile phone 12 is on the left side of computer 11 and tablet 13 is on the right side, computer 11 determines that the traversal direction for mobile phone 12 is the left side (i.e., if the mouse on computer 11 is slid to the left, the mouse pointer will reach the screen of mobile phone 12), and the traversal direction for tablet 13 is the right side (i.e., if the mouse on computer 11 is slid to the right, the mouse pointer will reach the screen of tablet 13).

[0228] As an example of automatically establishing a keyboard and mouse connection, if the first option in Figure 5 is selected, when the computer 11 detects that the mouse pointer touches the side edge of the computer screen twice in a row, the computer 11 can automatically connect to the device that meets the keyboard and mouse crossing conditions (such as connecting to the same network and / or logging into the same account). For example, after executing the above S403, if the communication module of the computer 11 detects an online Android device, S405 is executed. If the communication module of the computer 11 detects multiple online Android devices, it can establish a keyboard and mouse connection with each online Android device separately, or automatically select one or more of the devices to establish a keyboard and mouse connection.

[0229] In the mode of automatically establishing a keyboard and mouse connection, the PC may automatically set the traversal direction of the connected device, or the user may manually set the traversal direction of the connected device.

[0230] As an example of automatically setting the traversal direction, when the user turns on the option of automatically identifying the orientation, the PC can detect the position of the device with a keyboard and mouse connection through ultrasound, and automatically set the traversal direction of the device with a keyboard and mouse connection based on the detected position.

[0231] For example, in the interface shown in FIG5 , if the user checks or selects the option to “Automatically adjust screen arrangement as device placement changes,” the PC detects the user’s check or selection operation and, in response to this operation, can automatically identify the location of the currently connected device. For example, in the application scenario shown in FIG1 , when the user turns on the option to automatically identify the orientation, the mobile phone 12 or tablet 13 can detect its own motion using a motion sensor. When the motion stops, the computer 11 calculates the position of the mobile phone 12 or tablet 13 relative to the mobile phone 12 or tablet 13 using data obtained by the ultrasonic sensor and transmits the position to the computer 11. Upon receiving this position, the computer 11 can determine that the mobile phone 12 is on the left side of the computer 11 and the tablet 13 is on the right side of the computer 11. The computer 11 then sets the traversal direction of the mobile phone 12 to the left (i.e., when the mouse in the computer 11 is slid to the left, the mouse pointer can reach the screen of the mobile phone 12), and the computer 11 sets the traversal direction of the tablet 13 to the right (i.e., when the mouse in the computer 11 is slid to the right, the mouse pointer can reach the screen of the tablet 13).

[0232] It is understandable that when the user turns on the option of automatic orientation identification, if the position of the device to which the PC currently has a keyboard and mouse connection changes, the PC can detect the position change of the device to which the keyboard and mouse connection has been established through ultrasound and automatically adjust the crossing direction of the connected device.

[0233] It should be noted that the keyboard and mouse can not only move to the left or right side of the screen, but also move to the top or bottom of the screen, which is not limited in the embodiments of the present application.

[0234] In one possible implementation, the screen of the terminal device can be divided into the left side, the right side, and the middle. For example, the computer 11 can establish a coordinate system with the center of the screen as the origin, the parallel lines of the long side of the screen, and the parallel lines of the wide side of the screen, so as to facilitate the division of the left area, the right area, and the middle area. If the computer 11 detects that the mobile phone 12 and / or the tablet 13 are in the left area, it can be determined that the mobile phone 12 and / or the tablet 13 are on the left side of the computer 11. If the computer 11 detects that the mobile phone 12 and / or the tablet 13 are in the right area, it can be determined that the mobile phone 12 and / or the tablet 13 are on the right side of the computer 11. If the computer 11 detects that the mobile phone 12 and / or the tablet 13 are in the middle area, it can be determined that the mobile phone 12 and / or the tablet 13 are in the middle of the computer 11.

[0235] For example, see Figure 6, which is a position diagram provided by an embodiment of the present application. As shown in Figure 6, the computer 11 sets the area between (-45°, 45°) in the coordinate system as the right area, the area between (45°, 135°) and (-45°, -135°) as the middle area, and the area between (-135°, 135°) as the left area. If the angle of the mobile phone 12 relative to the computer 11 is between (-45°, 45°), the computer 11 detects that the mobile phone 12 is in the right area and can determine that the mobile phone 12 is on the right side of the computer 11.

[0236] In another possible implementation, the screen of the terminal device can be divided into the left side, the right side, the top and the bottom. For example, see Figure 7, which is a position diagram provided by another embodiment of the present application. As shown in Figure 7, the computer 11 sets the area between (-45°, 45°) in the coordinate system as the right area, the area between (45°, 135°) as the top area, the area between (-45°, -135°) as the bottom area, and the area between (-135°, 135°) as the left area. If the angle of the mobile phone 12 relative to the computer 11 is between (45°, 135°), the computer 11 detects that the mobile phone 12 is in the upper area, and it can be determined that the mobile phone 12 is above the computer 11.

[0237] It should be noted that the angle range of the area division shown in Figures 6 and 7 is only an example. The angle range of each area can be adjusted according to the user's experience. For example, the angle range of the middle area can be adjusted from (45°, 135°) to (30°, 150°), and from (-45°, -135°) to (-30°, -150°) according to the user's experience.

[0238] As an example of manually setting the traversal direction, when a user manually sets the traversal direction for a connected device, the PC may display a bubble at the edge of the screen when the mouse pointer is at the edge of the screen to indicate the presence of a traversable device (a device with a keyboard and mouse connection established). When the user selects a traversable device in the bubble, the PC sets the traversal direction for the selected device based on the current mouse pointer position on the screen.

[0239] For example, see Figure 8, which is a schematic diagram of a keyboard and mouse traversal system provided by an embodiment of the present application. As shown in Figure 8, the traversable devices of computer 11 include mobile phone 12 and tablet 13. When computer 11 detects that the mouse pointer is at the right edge of the computer screen, a bubble 801 is displayed on the right edge of the computer screen to indicate the presence of a traversable device. Bubble 801 also displays an icon 8011 for mobile phone 12 and an icon 8012 for tablet 13.

[0240] The user can operate the mouse to control the mouse pointer to click on the icon 8011 of the mobile phone 12. In response to the user operation, the computer 11 sets the crossing direction of the mobile phone 12 to the right, that is, slide the mouse in the computer 11 to the right, and the mouse pointer can reach the screen of the mobile phone 12.

[0241] The user can also operate the mouse to control the mouse pointer to click the icon 8012 of the tablet 13. In response to the user operation, the computer 11 sets the crossing direction of the tablet 13 to the right, that is, the mouse of the computer 11 slides to the right, and the mouse can reach the screen of the tablet 13.

[0242] Among them, the bubble box is only an example of a name, and the embodiments of the present application do not limit this. The number of traversable devices (devices that have formed a trust loop with the PC) can be one or more, and the embodiments of the present application do not limit this. When the number of traversable devices is one, the number of bubble boxes can be one. The bubble box can display the icon and model of the traversable device. For example: if the traversable device is a mobile phone, the mobile phone icon and the model of the mobile phone can be displayed, or the icon and / or model of the traversable device may not be displayed. The embodiments of the present application do not limit this.

[0243] When there are multiple traversable devices, the number of bubble boxes can be one, or the same as the number of traversable devices, and this embodiment of the present application does not limit this. If there are multiple traversable devices, the number of bubble boxes is multiple, each of the multiple bubble boxes can correspond to a traversable device, and each bubble box can display the icon and / or model of the corresponding traversable device. The user can click on the bubble box to select a traversable device from multiple traversable devices for keyboard and mouse traversal, or can select a traversable device from multiple traversable devices for keyboard and mouse traversal by clicking on the icon or model of the traversable device. This embodiment of the present application does not limit this. If there are multiple traversable devices, the number of bubble boxes is one, and this bubble box can display the icons and / or models of multiple traversable devices. The user can select a traversable device from multiple traversable devices for keyboard and mouse traversal by clicking on the icon or model of the traversable device.

[0244] Figure 8 only shows an example of a bubble box. For the case where the bubble box does not display the icon and / or model of the traversable device, or the case where multiple bubble boxes are displayed when there are multiple traversable devices, as well as other display forms of the bubble box, the embodiments of the present application will not provide additional examples.

[0245] In one possible implementation, each time the computer 11 detects that the mouse pointer is at the edge of the computer screen, a bubble box corresponding to the current traversable device is displayed.

[0246] In another possible implementation, when the computer 11 first detects that the mouse pointer is at the edge side1 of the computer screen, the bubble box corresponding to the current traversable device is displayed. After the user selects a traversable device, when the computer 11 detects that the mouse pointer is at the edge side1 of the computer screen again, the bubble box corresponding to the current traversable device is no longer displayed; and when the computer 11 detects that the mouse pointer is at the edge side2 of the computer screen, the bubble box corresponding to the current traversable device is displayed. In this implementation, when the computer 11 detects that the mouse pointer is at the edge side2 of the computer screen, the bubble box corresponding to the current traversable device can be displayed, or only the bubble box corresponding to the traversable device that is not currently selected can be displayed. For example, the current traversable devices are device 1, device 2, and device 3. If the mouse pointer of computer 11 is at the right edge of the computer screen and the user selects the bubble box corresponding to device 1, it means that the user has set device 1 as the crossing device on the right side of computer 11, that is, by sliding the mouse of computer 11 to the right, the mouse can reach the screen of device 1; when the mouse pointer of computer 11 is at the left edge of the computer screen, computer 11 displays the bubble boxes corresponding to device 2 and device 3 on the left edge of the computer screen for the user to select; if the user selects device 2, it means that the user has set device 2 as the crossing device on the left side of computer 11, that is, by sliding the mouse of computer 11 to the left, the mouse can reach the screen of device 2.

[0247] It should be noted that the crossing direction of the keyboard and mouse may be related to the position between the devices or may be unrelated to the position between the devices. This is not limited in the embodiments of the present application. For example, in the case where the crossing direction of the keyboard and mouse is related to the position between the devices, if device 1 is on the left side of device 2, the mouse of device 2 slides to the left, and the mouse can reach the screen of device 1. In the case where the crossing direction of the keyboard and mouse is unrelated to the position between the devices, if device 1 is on the left side of device 2, but the crossing direction of device 1 can be set to the right side of device 2 by manually establishing communication or manually setting the crossing direction, that is, the mouse of device 2 slides to the right, the mouse can reach the screen of device 1.

[0248] When a keyboard and mouse connection has been established between terminal devices, keyboard and mouse traversal between terminal devices can be achieved. The following will introduce the PC-centered keyboard and mouse traversal method. Among them, in the PC-centered keyboard and mouse traversal method, the keyboard and mouse connection is usually initiated by the PC, and the PC is the communication center. The PC can be connected to one or more traversable Android devices. In other words, the keyboard and mouse traversal between devices must go through the PC. The PC-centered keyboard and mouse traversal method can be applied to the communication system shown in Figure 1. As shown in Figure 1, the computer 11 is connected to the mobile phone 12 and the tablet 13 respectively. At this time, the computer 11 is the communication center for the mobile phone 12 and the tablet 13.

[0249] In the embodiment of the present application, in a PC-centric traversal mode, when a keyboard and mouse traversal event is initiated by computer 11 and the keyboard and mouse of computer 11 traverse to mobile phone 12 or tablet 13, computer 11 can be referred to as the source device or initiator, and mobile phone 12 or tablet 13 can be referred to as the peer device or receiver. When a keyboard and mouse traversal event is initiated by mobile phone 12 and the keyboard and mouse of mobile phone 12 traverse to computer 11 or tablet 13, mobile phone 12 can be referred to as the source device or initiator, and computer 11 or tablet 13 can be referred to as the peer device or receiver. However, the embodiment of the present application is not limited to this.

[0250] PC-centric keyboard and mouse crossover can be divided into two types: In the first crossover type, the operation task is mainly on the PC side, and the operation needs to be performed on the Android device. In this method, the PC keyboard and mouse can be crossovered to the Android device. For example, the keyboard and mouse of computer 11 in Figure 1 crossovers to the screen of mobile phone 12 or tablet 13. In the second crossover type, the operation task is mainly on the Android device side, and the operation needs to be performed on the PC. In this method, the keyboard and mouse of the Android device can be crossovered to the PC. For example, the keyboard and mouse of mobile phone 12 or tablet 13 in Figure 1 crossovers to the screen of computer 11.

[0251] The following introduces the two ways of crossing.

[0252] As an example of the first traversal method (traversal of a PC keyboard and mouse to an Android device), see Figure 9, which is a schematic diagram of a keyboard and mouse traversal scenario provided by an embodiment of the present application. The keyboard and mouse traversal scenario shown in Figure 9 can be applied to the communication system shown in Figure 1. As shown in Figure 1, a mobile phone 12 is connected to the right side of a computer 11 and a tablet 13 is connected to the left side.

[0253] As shown in FIG9 (a), when a user manipulates the mouse of computer 11 to move the mouse pointer 111 on the screen of computer 11 to the right, as shown in FIG9 (a) and (b), computer 11 can achieve keyboard and mouse crossover with mobile phone 12. In other words, the mouse pointer 111 displayed on computer 11 can cross from the screen of computer 11 to the right to the screen of mobile phone 12. When the mouse pointer 111 of computer 11 crosses to the screen of mobile phone 12, as shown in FIG9 (b), the mouse pointer 111 can be displayed as a circle on the screen of mobile phone 12. As shown in FIG9 (b), the user can also manipulate the mouse of computer 11 to move the mouse pointer 111 on the screen of mobile phone 12 to the left, as shown in FIG9 (b) and (c), the mouse pointer 111 on the screen of mobile phone 12 crosses back to the screen of computer 11 to the left, as shown in FIG9 (c), the mouse pointer 111 is displayed as an arrow on the screen of computer 11.

[0254] In some implementations, when the mouse of computer 11 is transferred to the screens of different Android devices, the mouse pointer 111 can be displayed in the same form on the screens of different Android devices. For example, in the communication system shown in FIG1 , computer 11 can also implement keyboard and mouse transfer with tablet 13. In other words, the mouse pointer 111 displayed on computer 11 can transfer from the screen of computer 11 to the screen of tablet 13 to the left. When the mouse pointer 111 of computer 11 transfers to the screen of tablet 13, the mouse pointer 111 can also be displayed as a circle on the screen of tablet 13.

[0255] In other implementations, when the mouse of the computer 11 is transferred to the screen of different Android devices, the mouse pointer 111 may be displayed in different forms on the screen of different Android devices. For example, when the mouse pointer 111 of the computer 11 is transferred to the screen of the tablet 13, the mouse pointer 111 may be displayed in the form of a diamond on the screen of the tablet 13.

[0256] In this example, the external mouse connected to computer 11 is displayed as an arrow on the screen of computer 11, while the external mouse connected to mobile phone 12 is displayed as a circle on the screen of mobile phone 12. When the mouse pointer 111 passes from the screen of computer 11 to the screen of mobile phone 12, it is displayed as a circle on the screen of mobile phone 12. When the mouse pointer 111 passes from the screen of mobile phone 12 back to the screen of computer 11, it is displayed as an arrow again on the screen of computer 11. It should be noted that in other examples, the mouse pointer can also be displayed in other forms after passing from the computer to the mobile phone, and this is not limited here.

[0257] After the keyboard and mouse of the computer 11 are transferred to the mobile phone 12 or tablet 13, the user can use the mouse to click, drag or double-click the mobile phone 12 or tablet 13, and can use the keyboard to input information on the mobile phone 12 or tablet 13.

[0258] Refer to Figure 10, which is a schematic diagram of the interactive scenario provided by an embodiment of the present application. As shown in the example in Figure 10, after the mouse pointer 111 of the computer 11 crosses to the screen of the mobile phone 12, the user can move the position of the mouse pointer 111 on the mobile phone screen by operating the mouse. As shown in (a) in Figure 10, the mouse pointer 111 is located on the "Memo" application icon. The user can click on the "Memo" application icon with the mouse pointer 111, and the mobile phone 12 can display the interface shown in (b) in Figure 10 in response to the user's operation of clicking the "Memo" application icon. As shown in (b) in Figure 10, the mobile phone 12 displays the memo interface. The user can enter the information "Memo Information" on the memo interface through the keyboard, and the mobile phone 12 displays the text "Memo Information" on the memo interface in response to the user's operation of entering information on the memo interface.

[0259] In order to better understand the embodiments of the present application, the implementation principle of the first crossing method (crossing the PC keyboard and mouse to the Android device) is introduced below in combination with the software architecture shown in Figures 2 and 3.

[0260] See Figure 11, which is a schematic diagram of the implementation principle of a traversal method provided by an embodiment of the present application. As shown in Figure 11, the implementation principle of traversal of a PC's keyboard and mouse to an Android device may include the following steps:

[0261] Step 000: The input driver of the Windows system detects the keyboard and mouse events of the PC and reports the keyboard and mouse events of the Windows system to the input framework of Windows.

[0262] Step 001: The input framework of the Windows system reports the received keyboard and mouse events to the keyboard and mouse service module of the Windows system.

[0263] In an embodiment of the present application, the input framework of the Windows system can obtain keyboard and mouse events, which include keyboard and mouse displacement (or position offset). The input framework of the Windows system reports the keyboard and mouse events to the keyboard and mouse service module of Windows. The keyboard and mouse service module of Windows determines whether keyboard and mouse crossing has occurred based on the keyboard and mouse displacement.

[0264] Step 002: If the keyboard and mouse service module of the Windows system identifies that a keyboard and mouse crossover has occurred (i.e., the keyboard and mouse of the PC crosses to the Android device) based on the received keyboard and mouse events, an instruction message of intercepting the event is sent to the input framework of the Windows system.

[0265] Intercept events are used to intercept keyboard and mouse events. The intercept event indication information is used to instruct the input framework to set up keyboard hooks, mouse hooks, and shortcut hooks. Among them, a hook is a platform for the Windows message processing mechanism. Applications can set up subroutines on it to monitor certain messages of a specified window, and the monitored window can be created by other processes. When a message arrives, it is processed before the target window processing function. The hook mechanism allows applications to intercept and process window messages or specific events. A hook is actually a program segment that processes messages and is hooked into the system through system calls. Whenever a specific message is sent, before it reaches the destination window, the hook program captures the message first, that is, the hook function obtains control first. At this time, the hook function can process (change) the message, or continue to pass the message without processing it, or force the message transmission to end.

[0266] The keyboard hook is used to obtain keyboard input operations, for example, the keyboard hook is used to obtain the user's keyboard input "memo information" operation. The mouse hook is used to obtain the user's mouse control operations, for example, the mouse hook is used to obtain the user's mouse click or drag operation. The shortcut key hook is used to obtain the user's shortcut key operation, for example, the shortcut key hook is used to obtain the user's Ctrl+C, Ctrl+V, ATL+TAB, or Ctrl+Alt+DELETE operation.

[0267] By intercepting events, the mouse passes from the PC to the Android device. The Windows system input framework intercepts the monitored keyboard and mouse events, making the business on the PC unresponsive to keyboard and mouse events, so that the PC does not affect the mouse events.

[0268] It should be noted that the input framework of the Windows system has the ability to distribute events and input data. Among them, data input refers to the Windows input framework reporting the relevant data of the event to the business module of the application layer. Event distribution refers to the Windows input framework reporting the event to the business module of the application layer. For the application layer, if a business module connected to the application layer receives the event reported by the input framework and the relevant data of the event, it can respond to the event according to the relevant data of the event. For example, the relevant data of the keyboard and mouse event includes the keyboard and mouse displacement. After a business module of the application layer responds to the keyboard and mouse event, it displays the movement of the keyboard and mouse and the position reached on the PC display screen according to the keyboard and mouse displacement. If a business module of the application layer receives the relevant data of the event but does not receive the event, it cannot respond to the event. In other words, intercepting the event is equivalent to notifying the various business modules of the application layer of the Windows system not to respond to the keyboard and mouse event, but does not affect the reporting of the keyboard and mouse event data (keyboard and mouse displacement).

[0269] Step 003: The keyboard and mouse service module of the Windows system packages the keyboard and mouse events according to a preset protocol to obtain a data packet of the keyboard and mouse events.

[0270] The keyboard and mouse events are packaged according to a preset protocol, and the relevant data involved in the keyboard and mouse events are converted into data in a corresponding format according to a data format defined in the preset protocol, and then the data in the converted format are packaged.

[0271] Optionally, the keyboard and mouse service module of the Windows system can encrypt data packets to prevent keyboard and mouse events from being leaked.

[0272] Step 004: The keyboard and mouse service module of the Windows system sends the data packet of the keyboard and mouse event to the communication module of the Windows system.

[0273] Optionally, the keyboard and mouse service module of the Windows system can send the keyboard and mouse event data packet and keyboard and mouse pass-through instruction information to the communication module of the Windows system. The keyboard and mouse pass-through instruction information is used to instruct the keyboard and mouse service module of the Android system to perform keyboard and mouse pass-through.

[0274] Step 005: The communication module of the Windows system sends the data packet of the keyboard and mouse events to the communication module of the Android system.

[0275] Step 006: The communication module of the Android system reports the received data packet to the keyboard and mouse service module of the Android system.

[0276] Step 007: The keyboard and mouse service module of the Android system parses the data packet according to a preset protocol to obtain keyboard and mouse events.

[0277] As described in step 003, if the keyboard and mouse service module of the Windows system encrypts the data packet, the keyboard and mouse service module of the Android system first decrypts the encrypted data packet to obtain the data packet, and then parses the data packet according to the preset protocol to obtain the keyboard and mouse event. The data obtained by the keyboard and mouse service module of the Android system includes the keyboard and mouse event and its related data (such as keyboard and mouse displacement).

[0278] It should be noted that the default protocol used by the keyboard and mouse service module of the Android system for parsing is the same as the default protocol used by the keyboard and mouse service module of the Windows system for packaging. In some application scenarios, the default protocol can be shared after the keyboard and mouse connection between the PC and the Android device is established.

[0279] Step 008: The keyboard and mouse service module of the Android system injects the keyboard and mouse events into the virtual input driver of the Android system.

[0280] Step 009: The virtual input driver of the Android system reports the keyboard and mouse events to the input framework of the Android system.

[0281] The input framework of the Android system can report keyboard and mouse events to various business modules in the Android application layer to implement keyboard and mouse functions.

[0282] Through the above steps, keyboard and mouse can be passed between PC and Android device.

[0283] It should be noted that the premise of achieving keyboard and mouse crossover between PC and Android device is that a keyboard and mouse connection has been established between PC and Android device. The method of establishing a keyboard and mouse connection is described in detail in the embodiment of Figure 4, which will not be repeated here.

[0284] In order to simplify the steps, the process of data interaction between the terminal devices by the communication module is omitted in the following embodiments, and the process of data interaction between the keyboard and mouse management submodule and the communication module by the connection scheduling submodule. But it is understandable that the data interaction between the keyboard and mouse business module of PC and the keyboard and mouse business module of Android device (such as the data packet of interactive keyboard and mouse event) can be understood as the keyboard and mouse business module of PC first transmitting data to the communication module of PC, then being transmitted to the communication module of Android device by the communication module of PC, and being transmitted to the keyboard and mouse business module of Android device by the communication module of Android device. In addition, the keyboard and mouse business module is described as a whole in the following embodiments, but it is understandable that the interaction between the keyboard and mouse business module and the communication module all needs to pass through the connection scheduling submodule in the keyboard and mouse business module, i.e., other submodules except the connection scheduling submodule in the keyboard and mouse business module first transmit data to the connection scheduling submodule, then being transmitted to the communication module by the connection scheduling submodule, or, the communication module first transmits data to the connection scheduling submodule, then being transmitted to other submodules in the keyboard and mouse business module by the connection scheduling submodule.

[0285] Based on the above implementation principles, the following introduces the first method of crossing (crossing the PC keyboard and mouse to the Android device).

[0286] See Figure 12, which is a schematic diagram of the interaction process of a keyboard and mouse traversal method provided by an embodiment of the present application. As shown in Figure 12, the interaction process of the keyboard and mouse traversal from a PC to an Android device may include the following steps:

[0287] S1201: Establish a keyboard and mouse connection between the PC and the Android device.

[0288] The keyboard and mouse connection in this step can be referred to the description in the embodiment of FIG4 , which will not be repeated here.

[0289] S1202: The keyboard and mouse service module of the PC registers a monitoring event of the keyboard and mouse events with the input framework of the PC.

[0290] After the keyboard and mouse service module of the PC registers the monitoring event of the keyboard and mouse time, it can monitor the keyboard and mouse events.

[0291] S1203: After registering the monitoring event, the input framework of the PC continuously reports the keyboard and mouse displacement to the keyboard and mouse service module of the PC.

[0292] For example, when a user presses the keyboard, moves, clicks, or drags the mouse, the PC's input framework can obtain the keyboard and mouse events and the keyboard and mouse displacement (or position offset) corresponding to the current keyboard and mouse events through the PC's input driver.

[0293] S1204: The keyboard and mouse service module of the PC determines whether the current mouse pointer reaches the edge of the PC screen based on the keyboard and mouse displacement.

[0294] It is understandable that S1203-S1204 can be performed continuously. In other words, after registering the monitoring event, the PC input framework continuously reports the keyboard and mouse displacement to the PC keyboard and mouse service module. Each time the PC keyboard and mouse service module receives the keyboard and mouse displacement, it determines whether the current keyboard and mouse have reached the edge of the PC screen based on the current keyboard and mouse displacement.

[0295] The process of determining the positions of the keyboard and mouse may refer to the description of the following S1301-S1305 embodiments.

[0296] S1205: If the current keyboard and mouse reach the edge of the PC screen, the keyboard and mouse service module of the PC obtains the crossing position of the keyboard and mouse.

[0297] If the current keyboard and mouse have not reached the edge of the PC screen, the keyboard and mouse service module of the PC continues to monitor the keyboard and mouse events.

[0298] In this step, the keyboard and mouse service module of the PC can determine the crossing position according to the keyboard and mouse displacement reported by the input framework.

[0299] S1206: Send a pass-through instruction to the keyboard and mouse service module of the Android device.

[0300] Correspondingly, the keyboard and mouse service module of the Android device receives the pass-through instruction. The keyboard and mouse service module of the PC notifies the keyboard and mouse service module of the Android device of the keyboard and mouse pass-through instruction.

[0301] S1207, after receiving the pass-through instruction, the keyboard and mouse service module of the Android device replies with a confirmation message to the keyboard and mouse service module of the PC.

[0302] S1208, the keyboard and mouse service module of the Android device determines the crossing position based on the crossing instruction.

[0303] In one implementation, the traverse instruction may include the traverse position of the keyboard and mouse on the PC screen. If the screen resolutions of the Android device and the PC are different, the mouse will appear in different positions on the screen when traversing between the two devices. The keyboard and mouse service module of the Android device can determine the traverse position of the mouse on the Android device screen based on the position of the mouse on the PC screen.

[0304] For example, if the mouse pointer's crossing position on the PC screen is (x1, y1) and the crossing direction is left crossing or right crossing, the PC's conversion module can be based on s y Change the transformation of y1, that is, y1×s y The value of the horizontal axis is a preset value, which is the minimum or maximum value of the screen side length on the X axis. R 1y is the side length of the y-axis of the Android device screen, R 2yThe length of the y-axis of the PC screen.

[0305] If the mouse's crossing position on the PC screen is (x1, y1) and the crossing direction is up or down, the PC's conversion module can be based on s x Change the transformation of x1, that is, x1×s x The value of the vertical coordinate is a preset value, which is the minimum or maximum value of the screen side length on the X axis. R 1x is the length of the x-axis of the Android device screen, R 2x The length of the x-axis of the PC screen.

[0306] For example, in the example shown in FIG13 , assuming that the screen resolution of the PC is 1920×1080, the crossing position is (1920,960), and the crossing direction is right crossing, the position after conversion can be (0,960×s y If the crossing position is (0,960) and the crossing direction is right crossing, and the screen resolution of the Android device is 2000×1000, the converted position can be (2000, 960×s y ).in,

[0307] It is understood that, in another implementation, the crossing instruction may include the proportion of the crossing position of the keyboard and mouse on the PC screen to the side length of the screen, that is, and The keyboard and mouse service module of the Android device calculates the keyboard and mouse crossing positions on the screen of the Android device according to the received ratio and the side length of the screen of the Android device.

[0308] It should be noted that the above x and s y This data can be calculated by the keyboard and mouse service module of the PC or Android device after the Android device establishes a keyboard and mouse connection with the PC. This data can be exchanged through the pass-through command.

[0309] S1209: The keyboard and mouse service module of the Android device updates the cross-travel status based on the cross-travel instruction.

[0310] Updating the crossing state in this step refers to updating the crossing state from not crossing to crossing.

[0311] S1210: After receiving the confirmation information, the keyboard and mouse service module of the PC sends an instruction message of intercepting the event to the input framework of the PC.

[0312] The interception event in this step has the same function as step 002 in the embodiment of FIG11 . Please refer to the description in the embodiment of FIG11 for details, which will not be repeated here.

[0313] It should be noted that S1208 - S1210 are not performed in any particular order and can be executed in parallel.

[0314] S1211: The keyboard and mouse service module of the PC packages the reported keyboard and mouse events according to a preset protocol to obtain a data packet of the keyboard and mouse events.

[0315] S1212: The keyboard and mouse service module of the PC sends the data packet to the keyboard and mouse service module of the Android device.

[0316] S1213: The keyboard and mouse service module of the Android device responds to the keyboard and mouse events.

[0317] Specifically, the keyboard and mouse service module of the Android device parses the data packet, obtains the keyboard and mouse displacement of the keyboard and mouse event, updates the keyboard and mouse position according to the keyboard and mouse displacement, and injects the keyboard and mouse event into the virtual input driver of the Android device according to the updated keyboard and mouse position; the virtual input driver of the Android device reports the keyboard and mouse event to the input framework of the Android device, and the input framework of the Android device distributes the keyboard and mouse event to various service modules of the application layer of the Android device to realize the function of PC keyboard and mouse on the Android device.

[0318] The principles of steps S1211-S1213 are the same as those of steps 003-008 in the embodiment of FIG11 . Please refer to the description in the embodiment of FIG11 for details, and are not separately illustrated in FIG12 .

[0319] S1214, the keyboard and mouse service module of the Android device monitors keyboard and mouse crossing.

[0320] In an embodiment of the present application, the process of the keyboard and mouse service module of the Android device monitoring the keyboard and mouse crossing includes: the keyboard and mouse service module of the Android device continuously receives the data packets sent by the keyboard and mouse service module of the PC, parses the keyboard and mouse events according to the data packets, and injects the keyboard and mouse events into the virtual input driver; the virtual input driver reports the keyboard and mouse events to the input framework of the Android device; the input framework reports the keyboard and mouse events to the various service modules of the application layer of the Android device, including the keyboard and mouse service module; the keyboard and mouse service module obtains the keyboard and mouse displacement according to the reported keyboard and mouse events.

[0321] It is understood that S1211-S1214 can be performed continuously while the PC's keyboard and mouse are being passed to the Android device. In other words, during this period, the PC's keyboard and mouse service module continuously sends data packets to the Android device's keyboard and mouse service module, and the Android device's keyboard and mouse service module continuously monitors the keyboard and mouse passthrough. Whenever a data packet is received, it performs corresponding processing based on the keyboard and mouse events parsed from the data packet.

[0322] Since the keyboard and mouse service module of the PC continuously sends data packets to the keyboard and mouse service module of the Android device, it can be more vividly understood here that the keyboard and mouse service module of the PC sends a data stream to the keyboard and mouse service module of the Android device.

[0323] After you've transferred your keyboard and mouse to your Android device, you can transfer it back. In this scenario, the keyboard and mouse transfer operation can be understood as the keyboard and mouse transfer back, that is, transferring your keyboard and mouse from your Android device to your PC. This is explained below.

[0324] S1215 , the keyboard and mouse service module of the Android device determines whether the current keyboard and mouse have reached the edge of the screen of the Android device based on the monitored keyboard and mouse displacement.

[0325] The process of determining the positions of the keyboard and mouse may refer to the description of the following S1301-S1305 embodiments.

[0326] S1216: If the current keyboard and mouse reach the edge of the screen of the Android device, a message to end the crossing is sent to the keyboard and mouse service module of the PC.

[0327] S1217, the keyboard and mouse service module of the PC receives the information of ending the pass-through, and sends a stop instruction to the input framework of the PC to stop intercepting the event.

[0328] The input framework of the PC receives the stop instruction of the interception event and stops intercepting the keyboard and mouse events. After stopping intercepting the keyboard and mouse events, the input framework of the PC redistributes the keyboard and mouse events to various business modules in the application layer of the PC.

[0329] S1218: The keyboard and mouse service module of the PC is restored.

[0330] In this step, restoring the state includes updating the crossing state and restoring the keyboard and mouse positions. Updating the crossing state means updating the crossing state from crossing to not crossing.

[0331] In one implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the PC can be restored based on the positions of the keyboard and mouse on the PC screen that are transferred back from the Android device.

[0332] In another implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the PC can be restored according to the positions of the keyboard and mouse that are crossed from the PC to the screen of the Android device.

[0333] In another implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the PC can be restored to a preset position, for example, the preset position is the center of the PC screen.

[0334] S1219, the keyboard and mouse service module of the PC sends a setting completion message to the keyboard and mouse service module of the Android device.

[0335] Correspondingly, the keyboard and mouse service module of the Android device receives the setting completion information, which indicates that the keyboard and mouse cross-connection has ended.

[0336] It should be noted that, in the embodiment of the present application, there is no limitation on the order of S1219 and S1217, and the two can be executed in parallel.

[0337] S1220: The keyboard and mouse service module of the Android device updates the traversal status based on the setting completion information.

[0338] In this step, updating the crossing state means updating the crossing state from crossing to not crossing.

[0339] The keyboard and mouse crossing method provided in the embodiment of the present application can adjust the keyboard and mouse crossing positions based on different devices after the keyboard and mouse cross-device crossing is achieved. After the mouse cross-device crossing is successful, the keyboard and mouse events are intercepted at the PC and responded to on the Android device side, thereby achieving keyboard and mouse cross-device crossing.

[0340] In the above method, S1204 and S1215 involve determining the position of the keyboard and mouse. The following describes a method for detecting the position of the mouse. The method for detecting the position of the mouse may include the following steps:

[0341] S1301. The keyboard and mouse service module obtains the current position of the mouse.

[0342] The current position of the mouse is the coordinate of the mouse on the screen. The keyboard and mouse service module can create a two-dimensional coordinate system on the screen. The coordinate of the mouse in the two-dimensional coordinate system is the current position of the mouse.

[0343] For example, FIG13 shows a schematic diagram of establishing a screen coordinate system. As shown in FIG13 , the screen OABC establishes a coordinate system with point O as the origin, the direction of OA as the positive direction of the X axis, and the direction of OC as the positive direction of the Y axis. The screen resolution is 1920×1080, the length of line segment OA is equal to the length of line segment CB (corresponding to 1920 pixels), and the length of line segment OC is equal to the length of line segment AB (corresponding to 1080 pixels). Then, in the coordinate system, the coordinates of point O are (0,0), the coordinates of point A are (1920,0), the coordinates of point B are (1920,1080), and the coordinates of point C are (0,1080). The mouse position is the coordinate of the mouse in the coordinate system. For example, the mouse position can be (1920,960).

[0344] S1302: The keyboard and mouse service module obtains the mouse displacement.

[0345] For example, when the user moves or drags the mouse, the PC's virtual input driver reports keyboard and mouse events to the PC's input framework. The PC's input framework obtains the mouse displacement based on the keyboard and mouse events and reports the mouse displacement to the PC's keyboard and mouse service module.

[0346] Mouse displacement is the distance the mouse moves. Mouse displacement can be represented by (Lx, Ly). Lx and Ly can be positive or negative values, and this embodiment of the application does not limit this.

[0347] S1303: The keyboard and mouse service module calculates the position of the mouse after it moves based on the mouse position and mouse displacement.

[0348] For example, the current position of the mouse can be represented by (XRaw, YRaw), and the position of the mouse after moving can be represented by (curX, curY). The position of the mouse after moving can be represented by the following formula: curX=XRaw+Lx; curY=YRaw+Ly.

[0349] S1304: The keyboard and mouse service module determines whether the position of the mouse after movement exceeds the screen.

[0350] In a scenario where the mouse moves across the screen left or right, if curX is less than the minimum X-axis value of the screen or greater than the maximum X-axis value of the screen, the PC's keyboard virtualization capability module can determine that the mouse is off-screen.

[0351] In a scenario where the mouse moves up and down the screen, if curY is less than the minimum value of the screen on the Y axis or greater than the maximum value of the screen on the Y axis, the PC's keyboard virtualization capability module can determine that the mouse is off the screen.

[0352] If the position of the mouse after moving exceeds the screen, it can be determined that the mouse has crossed, that is, execute S1305. If the position of the mouse after moving does not exceed the screen, the position of the mouse after moving can be determined as the current position, and continue to obtain the mouse movement, that is, execute S1302.

[0353] S1305: If the position of the mouse after movement exceeds the screen, the keyboard and mouse service module may determine that the mouse has crossed.

[0354] The keyboard and mouse service module can also determine the mouse's traversal direction. For example, if the mouse traverses the left and right sides of the screen, the traversal direction can be left or right. If the mouse traverses the top and bottom sides of the screen, the traversal direction can be up or down.

[0355] For example, in the example shown in FIG13 , if the mouse moves horizontally across the screen, and curX is less than 0, the keyboard and mouse service module can determine that the mouse's movement direction is left. If curX is greater than 1920, the keyboard and mouse service module can determine that the mouse's movement direction is right. If the mouse moves vertically across the screen, and curY is less than 0, the keyboard and mouse service module can determine that the mouse's movement direction is up. If curX is greater than 1080, the keyboard and mouse service module can determine that the mouse's movement direction is down.

[0356] It should be noted that the methods described in S1301-S1305 can be executed by a PC or an Android device. For example, in S1204, the methods in S1301-S1305 are executed by the keyboard and mouse service module of the PC. In S1215, the methods in S1301-S1305 are executed by the keyboard and mouse service module of the Android device.

[0357] The mouse crossing detection method provided in the embodiment of the present application can determine whether the mouse is beyond the screen and the crossing direction through the mouse position and mouse displacement, which is conducive to improving the accuracy of detecting mouse crossing.

[0358] As an example of the second traversal method (traversal of the keyboard and mouse of an Android device to a PC), see Figure 14, which is a schematic diagram of another traversal scenario provided by an embodiment of the present application. The traversal scenario shown in Figure 14 can be applied to the communication system shown in Figure 1. As shown in Figure 1, a mobile phone 12 is connected to the right side of a computer 11, and a tablet 13 is connected to the left side.

[0359] As shown in FIG14(a), when a user operates the mouse of mobile phone 12 to move the mouse pointer 121 on the screen of mobile phone 12 to the left, as shown in FIG14(a) and (b), keyboard and mouse crossover can be achieved between mobile phone 12 and computer 11. In other words, the mouse pointer 121 displayed on mobile phone 12 can cross from the screen of mobile phone 12 to the left to the screen of computer 11. When the mouse pointer 121 of mobile phone 12 crosses to the screen of computer 11, as shown in FIG14(b), the mouse pointer 121 can be displayed as an arrow on the screen of computer 11. As shown in FIG14(b), the user can also operate the mouse of mobile phone 12 to move the mouse pointer 121 on the screen of computer 11 to the right. As shown in FIG14(b) and (c), the mouse pointer 121 on the screen of computer 11 crosses back to the right to the screen of mobile phone 12. As shown in FIG14(c), the mouse pointer 121 is displayed as a circle on the screen of mobile phone 12.

[0360] In this example, the mouse pointer 121 is displayed as a circle on the mobile phone 12. The mouse pointer 121 moves from the mobile phone 12 to the computer 11, where it is displayed as an arrow on the computer 11. The mouse pointer 121 moves from the computer 11 back to the mobile phone 12, where it is again displayed as a circle on the mobile phone 12. When an external mouse is connected to the computer 11, the computer 11 also displays the mouse pointer as an arrow. In other examples, the mouse pointer may be displayed in other forms after moving from the mobile phone to the computer, which is not limited here.

[0361] After the keyboard and mouse of the mobile phone 12 are transferred to the computer 11, the user can use the mouse to click, drag or double-click the computer 11, and can use the keyboard to input information on the computer 11. This is not shown separately here.

[0362] In order to better understand the embodiments of the present application, the implementation principle of the second crossing method (crossing the keyboard and mouse of the Android device to the PC) is introduced below in combination with the software architecture shown in Figures 2 and 3.

[0363] See Figure 15, which is a schematic diagram of the implementation principle of a traversal method provided by an embodiment of the present application. As shown in Figure 15, the implementation principle of traversal of the keyboard and mouse of an Android device to a PC may include the following steps:

[0364] Step 010: The keyboard and mouse service module of the Windows system loads the virtual driver.

[0365] As shown in Figures 2 and 15, the Windows system's driver layer includes a virtual driver and an input driver. The virtual driver is used to drive the Android device's keyboard and mouse, while the input driver is used to drive the PC's keyboard and mouse. In this step, the Windows system's keyboard and mouse service module loads the virtual driver to provide driver support for subsequent Android device keyboard and mouse crossover to the PC.

[0366] As shown in FIG. 2 , the keyboard and mouse service module of the Windows system includes a virtual driver loading submodule. Accordingly, in step 010 , the virtual driver loading submodule of the Windows system can load the virtual driver.

[0367] It should be noted that step 010 may be a step executed after the keyboard and mouse connection between the PC and the Android device is established, or may be a step executed before step 019. This is not limited.

[0368] Step 011: The virtual input driver of the Android system detects the keyboard and mouse events of the Android device and reports the keyboard and mouse events of the Android system to the Android input framework.

[0369] Step 012: The input framework of the Android system reports the received keyboard and mouse events to the keyboard and mouse service module of the Android system.

[0370] In an embodiment of the present application, the input framework of the Android system can obtain the keyboard and mouse displacement (or position offset) corresponding to the keyboard and mouse event. The input framework of the Android system reports the keyboard and mouse event and the keyboard and mouse displacement corresponding to the keyboard and mouse event to the Android keyboard and mouse service module. The Android keyboard and mouse service module determines whether keyboard and mouse crossing has occurred based on the keyboard and mouse displacement.

[0371] Step 013, if the keyboard and mouse service module of the Android system recognizes that keyboard and mouse crossing has occurred (i.e., the keyboard and mouse of the Android device crosses to the PC screen) based on the received keyboard and mouse events, it calls the preset interface of the input framework of the Android system to intercept the keyboard and mouse event reporting.

[0372] Unlike Windows, Android intercepts keyboard and mouse events by calling the input framework's default APIs, rather than intercepting them. By calling these default APIs, the mouse passes from the Android device to the PC, where the Android input framework intercepts the detected keyboard and mouse events. This prevents services on the Android device from responding to these events, preventing the Android device from being affected by these events.

[0373] The principle / function of calling the preset interface of the input framework in the Android system is the same as the principle / function of the input framework executing interception events in the Windows system. For details, please refer to the description of step 002 in the embodiment of Figure 11, which will not be repeated here.

[0374] Step 014: The keyboard and mouse service module of the Android system packages and processes the keyboard and mouse events according to a preset protocol to obtain a data packet of the keyboard and mouse events.

[0375] Optionally, the keyboard and mouse service module of the Android system can encrypt data packets to prevent keyboard and mouse events from being leaked.

[0376] Step 015: The keyboard and mouse service module of the Android system sends the data packet of the keyboard and mouse event to the communication module of the Android system.

[0377] Optionally, the keyboard and mouse service module of the Android system can send the keyboard and mouse event data packet and keyboard and mouse pass-through instruction information to the communication module of the Android system. The keyboard and mouse pass-through instruction information is used to instruct the keyboard and mouse service module of the Windows system to execute keyboard and mouse pass-through.

[0378] Step 016: The communication module of the Android system sends the data packet of the keyboard and mouse events to the communication module of the Windows system.

[0379] Step 017: The communication module of the Windows system reports the received data packet to the keyboard and mouse service module of the Windows system.

[0380] Step 018: The keyboard and mouse service module of the Windows system parses the data packet according to the preset protocol to obtain keyboard and mouse events.

[0381] As described in step 014, if the keyboard and mouse service module of the Android system encrypts the data packet, the keyboard and mouse service module of the Windows system first decrypts the encrypted data packet to obtain the data packet, and then parses the data packet according to the preset protocol to obtain the keyboard and mouse event. The data parsed by the keyboard and mouse service module of the Windows system includes the keyboard and mouse event and its related data (such as keyboard and mouse displacement).

[0382] It should be noted that the default protocol used by the Windows keyboard and mouse service module for parsing is the same as the default protocol used by the Android keyboard and mouse service module for packaging. In some application scenarios, the default protocol can be shared after the PC and Android device establish a keyboard and mouse connection.

[0383] Step 019: The keyboard and mouse service module of the Windows system injects the keyboard and mouse events into the virtual driver of the Windows system.

[0384] Step 020: The virtual driver of the Windows system reports the keyboard and mouse events to the input framework of the Windows system.

[0385] The input framework of the Windows system can report keyboard and mouse events to various business modules in the application layer of Windows to implement keyboard and mouse functions.

[0386] Through the above steps, keyboard and mouse can be passed between Android devices and PC.

[0387] It should be noted that the premise of achieving keyboard and mouse traversal between the Android device and the PC is that a keyboard and mouse connection has been established between the PC and the Android device. The method of establishing a keyboard and mouse connection is described in detail in the embodiment of Figure 4 and will not be repeated here.

[0388] Based on the above implementation principles, the second method of traversal (traversal of the keyboard and mouse of an Android device to a PC) is described below. See Figure 16, which is a schematic diagram of the interaction process of a keyboard and mouse traversal method provided by an embodiment of the present application. As shown in Figure 16, the interaction process of traversal of the keyboard and mouse of an Android device to a PC may include the following steps:

[0389] S1601: Establish a keyboard and mouse connection between the PC and the Android device.

[0390] The keyboard and mouse connection in this step can be referred to the description in the embodiment of FIG4 , which will not be repeated here.

[0391] S1602: The keyboard and mouse service module of the Android device registers a listener event for keyboard and mouse events with the input framework of the Android device.

[0392] After the keyboard and mouse service module of the Android device registers the monitoring event of the keyboard and mouse time, it can monitor the keyboard and mouse events.

[0393] S1603: After registering the monitoring event, the input framework of the Android device continuously reports the keyboard and mouse displacement to the keyboard and mouse service module of the Android device.

[0394] For example, when a user performs a press operation on the keyboard, or moves, clicks, or drags the mouse, the input framework of the Android device can obtain the keyboard and mouse events and the keyboard and mouse displacement (or position offset) corresponding to the current keyboard and mouse events through the virtual input driver of the Android device.

[0395] S1604: The keyboard and mouse service module of the Android device determines whether the current mouse pointer reaches the edge of the screen of the Android device based on the keyboard and mouse displacement.

[0396] It is understandable that S1603-S1604 can be performed continuously. In other words, after registering the monitoring event, the input framework of the Android device continuously reports the keyboard and mouse displacement to the keyboard and mouse service module of the Android device. Each time the keyboard and mouse service module of the Android device receives the keyboard and mouse displacement, it determines whether the current keyboard and mouse have reached the edge of the Android device screen based on the current keyboard and mouse displacement.

[0397] The process of determining the positions of the keyboard and mouse can be found in the description of the method embodiment described in S1301-S1305 above, and will not be repeated here.

[0398] S1605: If the current keyboard and mouse reach the edge of the screen of the Android device, obtain the crossing position of the keyboard and mouse.

[0399] If the current keyboard and mouse have not reached the edge of the Android device's screen, the keyboard and mouse service module of the Android device continues to monitor keyboard and mouse events.

[0400] S1606: Send a pass-through instruction to the keyboard and mouse service module of the PC.

[0401] Correspondingly, the keyboard and mouse service module of the PC receives the pass-through instruction. The keyboard and mouse service module of the Android device notifies the keyboard and mouse service module of the PC of the keyboard and mouse pass-through through the pass-through instruction.

[0402] S1607, after receiving the cross-travel instruction, the keyboard and mouse service module of the PC replies with a confirmation message to the keyboard and mouse service module of the Android device if the current cross-travel status is not crossed.

[0403] S1608: The keyboard and mouse service module of the PC determines the crossing position based on the crossing instruction.

[0404] In this step, the method for the keyboard and mouse service module of the PC to determine the crossing position is the same as the principle of step S1208 in the embodiment of Figure 12. For details, please refer to the description in S1208 and will not be repeated here.

[0405] S1609: The keyboard and mouse service module of the PC updates the cross-travel status based on the cross-travel instruction.

[0406] Updating the crossing state in this step refers to updating the crossing state from not crossing to crossing.

[0407] S1610, after receiving the confirmation information, the keyboard and mouse service module of the Android device calls the preset interface of the input framework of the Android device to intercept the keyboard and mouse event reporting.

[0408] The preset interface in this step has the same function as step 013 in the embodiment of Figure 15. Please refer to the description in the embodiment of Figure 15 for details and will not be repeated here.

[0409] It should be noted that S1608 - S1610 are not performed in any particular order and can be executed in parallel.

[0410] S1611: The keyboard and mouse service module of the Android device packages the reported keyboard and mouse displacements according to a preset protocol to obtain a data packet of the keyboard and mouse event.

[0411] S1612: The keyboard and mouse service module of the Android device sends the data packet to the keyboard and mouse service module of the PC.

[0412] S1613: The keyboard and mouse service module of the PC responds to the keyboard and mouse events.

[0413] Specifically, the keyboard and mouse service module of the PC parses the data packet, obtains the keyboard and mouse displacement of the keyboard and mouse event, updates the keyboard and mouse position according to the keyboard and mouse displacement, and injects the keyboard and mouse event into the virtual driver of the PC according to the updated keyboard and mouse position; the virtual driver of the PC reports the keyboard and mouse event to the input framework of the PC, and the input framework of the PC distributes the keyboard and mouse event to various service modules of the application layer of the PC to realize the keyboard and mouse functions of the Android device on the PC.

[0414] The principles of steps S1611-S1613 are the same as those of steps 014-020 in the embodiment of FIG15 . Please refer to the description in the embodiment of FIG15 for details and will not be repeated here.

[0415] S1614, the PC's keyboard and mouse service module monitors keyboard and mouse crossing.

[0416] In an embodiment of the present application, the process of the PC's keyboard and mouse service module monitoring keyboard and mouse crossover includes: the PC's keyboard and mouse service module continuously receives data packets sent by the Android device's keyboard and mouse service module, parses the data packets to determine the keyboard and mouse displacement corresponding to the keyboard and mouse event, updates the keyboard and mouse position based on the keyboard and mouse displacement, and injects the updated keyboard and mouse position into the virtual driver to update the keyboard and mouse position on the PC screen. When the PC's keyboard and mouse service module monitors the keyboard and mouse crossover, it can obtain the keyboard and mouse displacement corresponding to the keyboard and mouse crossover event.

[0417] It is understood that during the period of keyboard and mouse passthrough from the Android device to the PC, S1611-S1614 can be performed continuously. In other words, during the period of keyboard and mouse passthrough from the Android device to the PC, the keyboard and mouse service module of the Android device continuously sends data packets to the keyboard and mouse service module of the PC, and the keyboard and mouse service module of the PC continuously monitors the keyboard and mouse passthrough. Whenever a data packet is received, it performs corresponding processing based on the keyboard and mouse events parsed in the data packet.

[0418] Since the keyboard and mouse service module of the Android device continuously sends data packets to the keyboard and mouse service module of the PC, it can be more vividly understood here that the keyboard and mouse service module of the Android device sends a data stream to the keyboard and mouse service module of the PC.

[0419] After you've transferred your keyboard and mouse to your PC, you can transfer it back. In the case of transferring your keyboard and mouse back to your Android device, the keyboard and mouse transfer operation can be understood as transferring your keyboard and mouse back, that is, transferring your keyboard and mouse from your PC back to your Android device. This is explained below.

[0420] S1615: The keyboard and mouse service module of the PC determines whether the current keyboard and mouse have reached the edge of the PC screen based on the monitored keyboard and mouse displacement.

[0421] The process of determining the positions of the keyboard and mouse can be found in the description of the method embodiment described in S1301-S1305 above, and will not be repeated here.

[0422] S1616: If the current keyboard and mouse reach the edge of the PC screen, a message to end the crossing is sent to the keyboard and mouse service module of the Android device.

[0423] S1617, the keyboard and mouse service module of the Android device receives the information of ending the crossing, stops calling the preset interface of the input framework of the Android device, and stops intercepting the keyboard and mouse event reporting.

[0424] After stopping calling the preset interface, the input framework of the Android device stops intercepting keyboard and mouse events. After stopping intercepting keyboard and mouse events, the input framework of the Android device redistributes keyboard and mouse events to various business modules in the application layer of the Android device.

[0425] S1618, the keyboard and mouse service module of the Android device has been restored.

[0426] In this step, restoring the state includes updating the crossing state and restoring the keyboard and mouse positions. Updating the crossing state means updating the crossing state from crossing to not crossing.

[0427] In one implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the Android device can be restored based on the positions of the keyboard and mouse on the screen of the Android device that are transferred from the PC back to the device.

[0428] In another implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the Android device can be restored according to the keyboard and mouse positions that are crossed from the Android device to the PC screen.

[0429] In another implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the Android device can be restored according to a preset position. For example, the mouse position is restored to the center position of the Android device screen.

[0430] S1619: The keyboard and mouse service module of the Android device sends a setting completion message to the keyboard and mouse service module of the PC.

[0431] Correspondingly, the keyboard and mouse service module of the PC receives the setting completion information, which is used to indicate that the keyboard and mouse cross-connection has ended.

[0432] It should be noted that, in the embodiment of the present application, there is no limitation on the order of S1619 and S1617, and the two can be executed in parallel.

[0433] S1620: The keyboard and mouse service module of the PC updates the traversal status based on the setting completion information.

[0434] In this step, updating the crossing state means updating the crossing state from crossing to not crossing.

[0435] The keyboard and mouse crossing method provided in the embodiment of the present application, when the keyboard and mouse achieve cross-device crossing, the keyboard and mouse crossing position can be adjusted based on different devices. After the mouse crossing is successful, the keyboard and mouse events are intercepted at the Android device and responded on the PC side, which can achieve keyboard and mouse cross-device crossing.

[0436] As shown in the embodiments of Figure 12 and Figure 16, the two are similar in that they both presuppose a keyboard and mouse connection has been established between the PC and the Android device, with the PC being the center of the keyboard and mouse connection. The difference between the two is that in Figure 12, the PC is the initiator of the keyboard and mouse crossover event, and the Android device is the receiver; in Figure 16, the Android device is the initiator of the keyboard and mouse crossover event, and the PC is the receiver.

[0437] Based on the two keyboard and mouse traversal methods described in the embodiments of Figures 12 to 16 above, users can flexibly operate multiple terminal devices through the same set of keyboard and mouse, avoiding switching operation modes back and forth between multiple terminal devices, simplifying operations, and helping to improve user experience.

[0438] The application scenarios shown in Figures 9 and 14 above are keyboard and mouse traversal scenarios between two terminal devices. In other application scenarios, a user may need to operate three or more terminal devices, which involves keyboard and mouse traversal between multiple terminal devices.

[0439] As an example of keyboard and mouse traversal between multiple terminal devices, keyboard and mouse traversal between three terminal devices is used as an example. Referring to Figures 17 and 18 , there are schematic diagrams of a keyboard and mouse traversal scenario provided by an embodiment of the present application. The keyboard and mouse traversal scenario shown in Figures 17 and 18 can be applied to the communication system shown in Figure 1 . As shown in Figure 1 , a mobile phone 12 is connected to the right side of a computer 11 and a tablet 13 is connected to the left side.

[0440] As shown in FIG17(a), when the user manipulates the mouse pointer on the screen of mobile phone 12 to move to the left, as shown in FIG17(a) and (b), keyboard and mouse crossover can be achieved between mobile phone 12 and computer 11. In other words, the mouse pointer 121 displayed on mobile phone 12 can cross from the screen of mobile phone 12 to the screen of computer 11. As shown in FIG17(b), when the user continues to manipulate the mouse of mobile phone 12 to move the mouse pointer on the screen of computer 11 to the left, as shown in FIG17(b) and (c), keyboard and mouse crossover can be achieved between computer 11 and tablet 13. In other words, the mouse pointer 121 displayed on computer 11 can continue to cross from the screen of computer 11 to the screen of tablet 13.

[0441] Continuing with the scenario shown in (c) of Figure 17 , the mouse pointer 121 is currently on the screen of tablet 13. As shown in (a) of Figure 18 , when the user manipulates the mouse pointer on the screen of tablet 13 to move rightward, as shown in (a) and (b) of Figure 18 , keyboard and mouse crossover can be achieved between tablet 13 and computer 11. In other words, the mouse pointer 121 displayed on tablet 13 can cross from the screen of tablet 13 to the right to the screen of computer 11. As shown in (b) of Figure 18 , when the user continues to manipulate the mouse pointer on the screen of computer 11 to move rightward, keyboard and mouse crossover can be achieved between computer 11 and mobile phone 12. In other words, the mouse pointer 121 displayed on computer 11 can cross from the screen of computer 11 back to the screen of mobile phone 12, as shown in (c) of Figure 18 .

[0442] In the application scenarios shown in FIG17 and FIG18 , the mouse pointer of the mobile phone 12 is sent multiple times across the computer 11 , the mobile phone 12 and the tablet 13 , thus realizing cross-PC keyboard and mouse crossing.

[0443] In a PC-centric application scenario where multiple Android devices implement keyboard and mouse passthrough across PCs, a keyboard and mouse event detected by Android device 1 may be a keyboard and mouse event of Android device 1 itself, in which case Android device 1 is in a non-passthrough state; or it may be a keyboard and mouse event of keyboard and mouse passthrough from Android device 2 or the PC, in which case Android device 1 is in a passthrough state. Failure to distinguish these events can lead to confusion in the passthrough state.

[0444] Based on this consideration, an embodiment of the present application provides a keyboard and mouse crossing method that can distinguish the keyboard and mouse crossing scenarios of multiple terminal devices, avoid the occurrence of confusion in the crossing status, and thus ensure the reliability of keyboard and mouse crossing.

[0445] In the application scenarios shown in Figures 17 and 18, multiple keyboard and mouse traversal messages are sent between a computer 11, a mobile phone 12, and a tablet 13. This can be divided into two scenarios: the first scenario involves the mouse pointer traversing from the PC screen to the Android device screen; the second scenario involves the mouse pointer traversing from the Android device screen to the PC screen. The following describes the keyboard and mouse traversal methods for each scenario.

[0446] It should be noted that in the application scenarios shown in Figures 17 and 18, keyboard and mouse crossover can occur between the PC and both Android devices. Therefore, the following embodiments assume that the PC has established keyboard and mouse connections with both Android devices. The method for establishing keyboard and mouse connections can be found in the description of the embodiment in Figure 4 above and will not be repeated here.

[0447] The following describes the keyboard and mouse traversal method for the first scenario. Referring to FIG19 , there is shown a schematic diagram of the interaction flow of a keyboard and mouse traversal method provided in an embodiment of the present application. The method described in FIG19 can be applied to a scenario where the mouse pointer traverses from a PC screen to an Android device screen. As an example and not a limitation, as shown in FIG19 , the keyboard and mouse traversal method may include the following steps:

[0448] S1901, the keyboard and mouse service module of Android device 1 monitors the keyboard and mouse displacement of the keyboard and mouse events.

[0449] The principle of this step is the same as that of S1214. For details, please refer to the description of S1214 and will not be repeated here.

[0450] S1902, the keyboard and mouse service module of Android device 1 determines whether the current mouse pointer reaches the edge of the screen based on the monitored keyboard and mouse displacement.

[0451] The method for determining whether the mouse pointer has reached the edge of the screen can be found in the description of the above S1301-S1305 embodiments, which will not be repeated here.

[0452] S1903: If the current mouse pointer reaches the edge of the screen, determine whether there is a traversable device at the edge of the current screen.

[0453] For example, if the Android device 1 is a tablet 13 as shown in FIG17 , since the Android device 1 is to the left of the PC, there are no traversable devices to the left of the Android device 1. Therefore, if the current mouse pointer reaches the left edge of the screen, keyboard and mouse traversal is not considered; if the current mouse pointer reaches the right edge of the screen, S1904 is executed. If the Android device 1 is a mobile phone 12 as shown in FIG17 , since the Android device 1 is to the right of the PC, there are no traversable devices to the right of the Android device 1. Therefore, if the current mouse pointer reaches the right edge of the screen, keyboard and mouse traversal is not considered; if the current mouse pointer reaches the left edge of the screen, S1904 is executed.

[0454] S1904: If there is a traversable device at the edge of the current screen, determine whether the current mouse pointer belongs to the local keyboard and mouse (the keyboard and mouse of Android device 1).

[0455] As can be seen from the application scenarios shown in Figures 17 and 18, the mouse pointer on the Android device screen may be the mouse pointer of its own keyboard and mouse, or it may be a mouse pointer passed through from a PC.

[0456] If the current mouse pointer belongs to the local keyboard and mouse (the keyboard and mouse of the Android device 1), it is the case of the keyboard and mouse of the Android device being crossed to the PC as shown in FIG14 . The keyboard and mouse crossing method can be referred to steps S1605-S1615 in the embodiment of FIG16 . No further details will be given here.

[0457] It should be noted that the application scenario shown in Figure 14 only involves keyboard and mouse passthrough between a PC and a single Android device. Unlike the application scenario shown in Figure 14, the application scenarios shown in Figures 17 and 18 involve keyboard and mouse passthrough between a PC and multiple Android devices. Therefore, before the keyboard and mouse of Android device 1 passthrough to the PC, the keyboard and mouse of the PC may have already passedthrough to Android device II. In this case, the PC is in a passthrough state, meaning that the PC's input framework is executing interception events to prevent keyboard and mouse event reporting.

[0458] In order to ensure that the PC can execute keyboard and mouse events after the keyboard and mouse of Android device I are passed through to the PC, in one embodiment, after S1606, the keyboard and mouse service module of the PC receives the passing instruction sent by Android device I, and then sends a stop instruction to the input framework of the PC to stop intercepting keyboard and mouse events, so as to instruct the input framework of the PC to stop intercepting keyboard and mouse events.

[0459] If the current mouse pointer does not belong to the local keyboard and mouse, it means that the current mouse pointer has passed through the screen of the PC, and then step S1905 is executed.

[0460] In some embodiments, during keyboard and mouse passthrough, the device ID of the device to which the keyboard and mouse belong can be packaged into a data packet along with the keyboard and mouse events. For example, if the keyboard and mouse of Android device 1 is passed through to a PC, the keyboard and mouse service module of Android device 1 packages the device ID of Android device 1 and the keyboard and mouse events into a data packet and sends it to the keyboard and mouse service module of the PC. The keyboard and mouse service module of the PC parses the received data packet to extract the keyboard and mouse events and the device ID. Based on the device ID, it can confirm that the current keyboard and mouse are passed through from Android device 1.

[0461] Accordingly, one implementation of determining whether it is a local keyboard and mouse may include:

[0462] The keyboard and mouse service module obtains the device ID corresponding to the current keyboard and mouse event; and determines whether it is the local keyboard and mouse based on the device ID.

[0463] It should be noted that when an Android device determines whether it is a local keyboard and mouse, the keyboard and mouse service module of the Android device performs the above determination step. When a PC device determines whether it is a local keyboard and mouse, the keyboard and mouse service module of the PC performs the above determination step.

[0464] In other implementations, during the keyboard and mouse pass-through process, the device ID can be carried in each transmitted keyboard and mouse event data packet; the device ID can also be carried in the first transmitted keyboard and mouse event data packet, while the device ID is no longer carried in the subsequent transmitted keyboard and mouse event data packets. In other implementations, the device ID can also be carried in the pass-through instruction sent when the keyboard and mouse pass-through occurs for the first time.

[0465] In other embodiments, the crossing direction of the keyboard and mouse can also be packaged into a data packet together with the keyboard and mouse events. For example, the keyboard and mouse of Android device 1 cross to the PC, and Android device 1 is on the left side of the PC, that is, the crossing direction is to the right. The keyboard and mouse service module of Android device 1 packages the crossing direction (to the right) and the keyboard and mouse events together into a data packet and sends it to the keyboard and mouse service module of the PC. The keyboard and mouse service module of the PC parses the keyboard and mouse events and the crossing direction (to the right) from the received data packet, and can confirm that the current keyboard and mouse are crossed from Android device 1 based on the crossing direction.

[0466] In the application scenario shown in Figures 17 and 18, the keyboard and mouse of Android device II first pass through to the PC and then pass through to Android device I. When the mouse pointer passes back from Android device I to the PC, if the judgment is based solely on the passage direction, it may be mistakenly judged that the current keyboard and mouse belong to Android device I. To prevent misjudgment, in other embodiments, the passage direction of the keyboard and mouse and the device ID to which the keyboard and mouse belong can also be packaged into a data packet along with the keyboard and mouse event. In this way, the terminal to which the keyboard and mouse belong can be determined not only by the passage direction but also by the device ID. This dual judgment can effectively prevent misjudgment and facilitate clearer distinction between different passage scenarios.

[0467] In other implementations, a preset function can be called to determine whether the keyboard and mouse are the user's own or virtual keyboard and mouse. For example, the preset function can return a function value. When the function value is false, it indicates a virtual keyboard and mouse; when the function value is true, it indicates the user's own keyboard and mouse.

[0468] S1905, if the current mouse pointer does not belong to the local keyboard and mouse (the keyboard and mouse of Android device 1), the keyboard and mouse service module of Android device 1 sends a return instruction (second instruction) to the keyboard and mouse service module of the PC and ends the crossing.

[0469] The keyboard and mouse service module of Android device 1 can end the crossing according to steps S1617-S1619.

[0470] Correspondingly, the keyboard and mouse service module of the PC receives the return instruction. The keyboard and mouse service module of the Android device 1 informs the keyboard and mouse service module of the PC through the return instruction that the mouse pointer passes back to the PC.

[0471] S1906: The keyboard and mouse service module of the PC receives the return instruction and determines whether the currently traversing mouse pointer belongs to the local keyboard and mouse (the keyboard and mouse of the PC).

[0472] The principle of determining whether it is a local keyboard and mouse in this step is the same as that in S1904. For details, please refer to the description in S1904 and will not be repeated here.

[0473] If the currently traversed mouse pointer belongs to the local keyboard and mouse (the keyboard and mouse of the PC), it is the case of the PC keyboard and mouse being traversed from the Android device back to the PC as shown in FIG9 . The keyboard and mouse traversal method can be referred to steps S1616-S1620 in the embodiment of FIG16 . No further details will be given here.

[0474] If the currently traversed mouse pointer does not belong to the local keyboard and mouse (the keyboard and mouse of the PC), it means that the current mouse pointer is traversed from the Android device II, and then S1907 is executed.

[0475] It should be noted that in the embodiment of the present application, when the PC's keyboard and mouse service module determines that the current mouse pointer does not belong to the local keyboard and mouse, the PC's keyboard and mouse service module can determine whether the currently traversing keyboard and mouse belongs to Android device I or Android device II based on the type of command received from Android device I. For example, if a traversal command is received, it indicates that the currently traversing keyboard and mouse belongs to Android device I; if a return command is received, it indicates that the currently traversing keyboard and mouse belongs to Android device II. Of course, a collaborative determination can also be made based on the device ID in the data packet.

[0476] S1907, the keyboard and mouse service module of the PC sends a switching notification (third instruction) to the keyboard and mouse service module of the Android device II.

[0477] S1908: The keyboard and mouse service module of the Android device II receives the switching notification and sends the data packet of the keyboard and mouse event to the keyboard and mouse service module of the PC.

[0478] Afterwards, the keyboard and mouse service module of the PC monitors the data packet of the keyboard and mouse event and executes steps S1613-S1615.

[0479] The situation corresponding to this step is the same as the situation in which the keyboard and mouse of the Android device are crossed to the PC as shown in Figure 14. This step has the same principle as S1613-S1615. For details, please refer to the description of S1613-S1615, which will not be repeated here.

[0480] It should be noted that, in order to simplify the steps, Figure 19 only shows part of the software architecture in the PC and Android device, which does not mean that other modules not shown do not participate in keyboard and mouse crossing. The modules involved in keyboard and mouse crossing can be found in the description in other embodiments.

[0481] For example, in some application scenarios, in the embodiment of FIG19 , a PC can be the second device, Android device I can be the first device, and Android device II can be the third device; or a PC can be the second device, Android device II can be the first device, and Android device I can be the third device. In this application scenario, the second device is communicatively connected to the first device and the third device, respectively.

[0482] In an embodiment of the present application, in an application scenario where keyboard and mouse traversal occurs between multiple terminal devices, when the mouse pointer traverses from Android device 1 to the PC screen, the Android device 1 side determines whether it is a local keyboard and mouse, thereby distinguishing whether the keyboard and mouse of Android device 1 traverses to the PC or the keyboard and mouse of Android device 1 traverses back to the PC; when the keyboard and mouse of Android device 1 traverses back to the PC, the PC side determines whether it is a local keyboard and mouse, thereby distinguishing whether the keyboard and mouse of Android device 1 traverses back to the PC or the keyboard and mouse of Android device II traverses to the PC. Through the above method, the keyboard and mouse traversal scenarios of multiple terminal devices can be distinguished, the occurrence of confusion in the traversal status can be avoided, and the reliability of keyboard and mouse traversal can be ensured.

[0483] The following describes the keyboard and mouse traversal method for the second scenario. Referring to FIG. 20 , there is shown a schematic diagram of the interaction flow of a keyboard and mouse traversal method provided in an embodiment of the present application. The method described in FIG. 20 can be applied in a scenario where the mouse pointer traverses from the screen of an Android device to the screen of a PC. As an example and not a limitation, as shown in FIG. 20 , the keyboard and mouse traversal method may include the following steps:

[0484] S2001, the keyboard and mouse service module of the PC monitors the keyboard and mouse displacement of the keyboard and mouse events.

[0485] The principle of this step is the same as that of S1614. For details, please refer to the description of S1614 and will not be repeated here.

[0486] S2002: The keyboard and mouse service module of the PC determines whether the current mouse pointer reaches the edge of the screen based on the monitored keyboard and mouse displacement.

[0487] The method for determining whether the mouse pointer has reached the edge of the screen can be found in the description of the above S1301-S1305 embodiments, which will not be repeated here.

[0488] S2003: If the current mouse pointer reaches the edge of the screen, determine whether there is a traversable device at the edge of the current screen.

[0489] For example, when there is a tablet 13 on the left side of the PC and no device is connected to the right side, when the current mouse pointer on the PC screen reaches the left edge of the screen, there is a traversable device, tablet 13, then S2004 is executed; when the current mouse pointer on the PC screen reaches the right edge of the screen, there is no traversable device, and there is no need to consider keyboard and mouse traversal at this time.

[0490] S2004: If there is a traversable device at the edge of the current screen, determine whether the current mouse pointer belongs to the local keyboard and mouse (the keyboard and mouse of the PC).

[0491] As can be seen from the application scenarios shown in Figures 17 and 18, the mouse pointer on the PC screen may be the mouse pointer of the native keyboard and mouse, or it may be the mouse pointer passed through from the Android device.

[0492] If the current mouse pointer belongs to the local keyboard and mouse (PC keyboard and mouse), it is the scene of the PC keyboard and mouse crossing to the Android device as shown in Figure 9. The keyboard and mouse crossing method can be found in steps S1205-S1215 in the embodiment of Figure 12, which will not be repeated here.

[0493] It should be noted that FIG20 illustrates the case where the mouse pointer moves from the left side of the PC screen to Android device 1; accordingly, steps S1205-S1215 are executed between the PC and Android device 1. It will be appreciated that if the mouse pointer moves from the right side of the PC screen to Android device II, steps S1205-S1215 are executed between the PC and Android device II.

[0494] If the current mouse pointer does not belong to the local keyboard and mouse (the keyboard and mouse of the PC), step S2005 is executed.

[0495] S2005: If the current mouse pointer does not belong to the local keyboard and mouse (the keyboard and mouse of the PC), determine whether the current mouse pointer belongs to the receiving end.

[0496] The receiving end represents the Android device corresponding to the current traversal direction. For example, in the communication system shown in Figure 1, if the current mouse pointer traverses from the screen of computer 11 to the left, and tablet 13 is to the left of computer 11, tablet 13 is the receiving end. If the current mouse pointer traverses from the screen of computer 11 to the right, and mobile phone 12 is to the right of computer 11, mobile phone 12 is the receiving end.

[0497] If the current mouse pointer belongs to the receiving end, that is, the receiving end is the source end, which is the case of the keyboard and mouse of the Android device being passed from the PC to the Android device as shown in Figure 14, the keyboard and mouse passing method can be referred to steps S1616-S1620 in the embodiment of Figure 16. No further details will be given here.

[0498] The source end refers to the Android device to which the current mouse pointer belongs. For example, in the communication system shown in Figure 1, if the current mouse pointer belongs to mobile phone 12, then mobile phone 12 is the source end, indicating that the keyboard and mouse of mobile phone 12 first pass through to computer 11, and then from computer 11 to tablet 13. If the current mouse pointer belongs to tablet 13, then tablet 13 is the source end, indicating that the keyboard and mouse of tablet 13 first pass through to computer 11, and then from computer 11 back to tablet 13.

[0499] If the current mouse pointer does not belong to the receiving end, it means that the current mouse pointer is passed from the Android device II, and then S2006 is executed.

[0500] It should be noted that FIG20 shows a situation where the mouse pointer passes from the left side of the PC screen to the Android device I. In this situation, the Android device I is the receiving end and the Android device II is the source end.

[0501] S2006: If the current mouse pointer does not belong to the receiving end (Android device I), a transfer instruction (first instruction) is sent to the source end (Android device II).

[0502] Accordingly, the source end can receive the transfer instruction, which is used to inform the source end that the receiving end of the keyboard and mouse events has changed.

[0503] This step describes the scenario where a keyboard and mouse are transferred from one Android device to another via a PC. For example, in the communication system shown in Figure 1, if the current mouse pointer on the screen of computer 11 is the keyboard and mouse of mobile phone 12 (the source), and the current mouse pointer transfers to the left to tablet 13 (the receiver), the receiving end of the keyboard and mouse transfer changes from computer 11 to tablet 13.

[0504] S2007, after receiving the transfer instruction, the keyboard and mouse service module of the source end (Android device II) sends a crossing instruction to the keyboard and mouse service module of the receiving end (Android device I).

[0505] In some implementations, the source end may send the traversal instruction to the receiving end through the PC, that is, the source end first sends the traversal instruction to the PC, which then forwards it to the receiving end.

[0506] In other implementations, the source end may request to directly establish a keyboard and mouse connection with the receiving end. After the keyboard and mouse are connected, the source end directly sends a pass-through instruction to the receiving end.

[0507] S2008, after the receiving end (Android device I) receives the crossing instruction, if the current crossing status is not crossed, it replies with a confirmation message to the keyboard and mouse service module of the source end (Android device II).

[0508] In some scenarios, if the receiving end is currently in the "travel in progress" state, it will reply to the keyboard and mouse service module on the source end with an "occupied" message to inform the source end that the receiving end is currently in the "travel in progress" state. In this case, the source end can notify the user on the screen that keyboard and mouse travel is not possible at this time.

[0509] It should be noted that, in the embodiment of the present application, if a device is currently in a crossing state, keyboard and mouse crossing will no longer be performed with other devices.

[0510] S2009, the keyboard and mouse service module of the receiving end (Android device 1) updates the crossing status based on the crossing instruction.

[0511] In this step, updating the crossing status means updating the crossing status from not crossing to crossing.

[0512] S2010, the keyboard and mouse service module of the receiving end (Android device 1) determines the crossing position based on the crossing instruction.

[0513] In this step, the method for the receiving end to determine the crossing position can be referred to the description of the embodiments S1301-S1305.

[0514] S2011, after receiving the confirmation information, the keyboard and mouse service module of the source end (Android device II) packages the keyboard and mouse events reported by the input framework of the source end (Android device II) according to the preset protocol to obtain a data packet of the keyboard and mouse events.

[0515] S2012, the source end (Android device II) sends the data packet of the keyboard and mouse event to the keyboard and mouse service module of the receiving end (Android device I).

[0516] In the embodiment of the present application, the communication system uses the PC as the communication center. In other words, all interactions between Android device I and Android device II must go through the PC. One implementation of this step is that the keyboard and mouse service module on the source side sends the data packet of the keyboard and mouse event to the communication module on the source side. The communication module on the source side sends the data packet to the communication module on the PC. The communication module on the PC forwards the data packet to the communication module on the receiving side. The communication module on the receiving side sends the data packet to the keyboard and mouse service module on the receiving side.

[0517] In order to improve communication efficiency, another implementation method of step S2012 is that the source end and the receiving end establish a communication connection through the PC; after the communication connection between the source end and the receiving end is established, the keyboard and mouse service module of the source end sends the data packet of the keyboard and mouse event to the communication module of the source end, and the communication module of the source end sends the data packet to the communication module of the receiving end; the communication module of the receiving end sends the data packet to the keyboard and mouse service module of the receiving end.

[0518] It should be noted that, in order to simplify the steps in the schematic diagram, S2012 in FIG20 omits the above-mentioned data interaction process.

[0519] S2013, the keyboard and mouse service module of the receiving end (Android device 1) responds to the keyboard and mouse events.

[0520] S2014, the keyboard and mouse service module of the receiving end (Android device 1) monitors the keyboard and mouse crossing.

[0521] The principles of steps S2013-S2014 are the same as those of S1213-S1214. For details, please refer to the description of S1213-S1214 and will not be repeated here.

[0522] In some scenarios, a communication failure may occur between the source and the receiver, resulting in a disconnection between the source and the receiver. In this scenario, S2015 is executed.

[0523] S2015: If the communication between the source end and the receiving end is disconnected, the source end and the receiving end respectively recover their states.

[0524] Among them, the source end recovery state includes the source end stopping sending data packets of keyboard and mouse events to the receiving end, restoring the mouse pointer of the source end keyboard and mouse to the first preset position on the source end screen, and maintaining the communication connection with the PC. For example, the first preset position can be the center position of the source end screen, or the position of the initial crossing on the source end screen. The receiving end recovery state includes restoring the mouse pointer of the receiving end keyboard and mouse to the second preset position on the receiving end screen, and maintaining the communication connection with the PC. For example, the second preset position can be the center position of the receiving end screen, or the position of the mouse pointer on the receiving end screen before the keyboard and mouse crossing occurs. In other words, after the recovery state, the mouse pointers of the respective keyboards and mice of the source end and the receiving end return to their respective screens.

[0525] It should be noted that, in order to simplify the steps, Figure 20 only shows part of the software architecture in the PC and Android device, which does not mean that other modules not shown do not participate in keyboard and mouse crossing. The modules participating in keyboard and mouse crossing can be found in the description in other embodiments.

[0526] It should be noted that Figure 20 shows the situation where Android device II is the source end and Android device I or PC is the receiving end. The principle is similar for the situation where Android device I is the source end and Android device II or PC is the receiving end, so it will not be repeated here.

[0527] For example, in some application scenarios, in the embodiment of FIG20 , a PC can be the second device, Android device I can be the first device, and Android device II can be the third device; or a PC can be the second device, Android device II can be the first device, and Android device I can be the third device. In this application scenario, the second device is communicatively connected to the first device and the third device, respectively.

[0528] In an embodiment of the present application, in an application scenario where keyboard and mouse pass-through occurs between multiple terminal devices, when the mouse pointer passes from the PC screen to the screen of Android device 1, the PC side determines whether the keyboard and mouse are local to the device. This allows the PC to distinguish whether the keyboard and mouse are passing from the PC to Android device 1, from the PC back to Android device 1, or from Android device II to Android device 1. This method distinguishes between keyboard and mouse pass-through scenarios across multiple terminal devices, avoids confusion in the pass-through state, and thus ensures the reliability of keyboard and mouse pass-through.

[0529] In the embodiment of the present application, after the mouse pointer crosses, the mouse pointer after the cross and the local mouse pointer can be combined into one mouse pointer, thereby avoiding confusion of the mouse pointers.

[0530] It should be noted that the above embodiments are all described with a PC as the communication center. In actual applications, an Android device can also be used as the center, that is, an Android device is connected to multiple terminal devices, and the Android device initiates the keyboard and mouse connection. In other words, the Android device is the initiator and the PC is the receiver. The keyboard and mouse traversal method with an Android device as the communication center is the same as the keyboard and mouse traversal method with a PC as the communication center, and will not be repeated in this embodiment.

[0531] The various embodiments described herein may be independent solutions or may be combined according to internal logic, and all of these solutions fall within the scope of protection of this application.

[0532] The above mainly describes the solution provided by the embodiment of the present application from the perspective of method steps. It is understandable that, in order to achieve the above functions, the electronic device implementing the method includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should be aware that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of protection of this application.

[0533] The present application also provides a chip, which is coupled to a memory and is used to read and execute computer programs or instructions stored in the memory to perform the methods in the above embodiments.

[0534] The present application also provides an electronic device, which includes a chip, and the chip is used to read and execute computer programs or instructions stored in a memory, so that the methods in each embodiment are executed.

[0535] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0536] This embodiment further provides a computer program product, wherein the computer-readable storage medium stores program code. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0537] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the gaze point estimation method in the above-mentioned method embodiments.

[0538] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0539] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0540] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A keyboard and mouse traversal method, characterized in that: The method comprises: In a case where a cursor of an input device is displayed on a screen of a first device, the first device detects a first operation applied to the input device, the first operation being used to move the cursor displayed on the screen of the first device in a first direction, wherein the input device is used to input a control instruction; In response to the first operation, the first device controls the cursor to move out of the screen of the first device and enter the screen of the second device for display, wherein the second device is located in the first direction of the first device and is connected to the first device; The second device detects a second operation applied to the input device, the second operation being used to move a cursor displayed on a screen of the second device in a second direction; In response to the second operation, the second device controls the cursor to move out of the screen of the second device and to be displayed on the screen of a third device, wherein the third device is located in the second direction of the second device and is connected to the second device.

2. The method according to claim 1, characterized in that In response to the first operation, the first device controls the cursor to move out of the screen of the first device and enter the screen of the second device for display, including: In response to the first operation, the first device finds a second device that establishes a connection with the first device in the first direction; The first device controls the cursor to move out of the screen of the first device and to be displayed on the screen of the second device.

3. The method according to claim 1 or 2, characterized in that The method further comprises: In response to the first operation, the first device determines whether the input device is connected to the first device; If the input device is connected to the first device, the first device updates the crossing state of the first device to crossing; The second device updates the crossing status of the second device to crossing.

4. The method according to claim 3, characterized in that After the first device determines, in response to the first operation, whether the input device is connected to the first device, the method further includes: If the input device is connected to the first device, the first device detects a third operation, and the third operation is used to input the control instruction; The first device intercepts a first input event corresponding to the third operation; The first device sends the first input event to the second device; After receiving the first input event, the second device executes the control instruction corresponding to the first input event.

5. The method according to any one of claims 1 to 4, characterized in that: In response to the second operation, the second device controls the cursor to move out of the screen of the second device and enter the screen of the third device for display, including: In response to the second operation, the second device searches for a third device that establishes a connection with the second device in the second direction; The second device controls the cursor to move out of the screen of the second device and to be displayed on the screen of the third device.

6. The method according to claim 3, characterized in that After the first device determines, in response to the first operation, whether the input device is connected to the first device, the method further includes: If the input device is connected to the first device, in response to a fourth operation, the second device determines whether the first device is the same as the third device, and the fourth operation is used to move a cursor displayed on a screen of the second device in the second direction; If the first device is the same as the third device, the first device updates the crossing status of the first device to not crossing; The second device updates the crossing status of the second device to not crossing.

7. The method according to claim 6, characterized in that After, in response to the fourth operation, the second device determines whether the first device is the same as the third device, the method further includes: If the first device is the same as the third device, the first device stops intercepting the first input event corresponding to the third operation; The first device executes the control instruction corresponding to the first input event.

8. The method according to claim 6, characterized in that After, in response to the fourth operation, the second device determines whether the first device is the same as the third device, the method further includes: If the first device is the same as the third device, the first device controls the cursor to be displayed at a first preset position on the screen of the first device.

9. The method according to claim 6, characterized in that After, in response to the fourth operation, the second device determines whether the first device is the same as the third device, the method further includes: If the first device is different from the third device, the second device sends a first instruction to the first device; After receiving the first instruction, the first device controls the cursor to move out of the screen of the second device and to be displayed on the screen of the third device.

10. The method according to claim 9, characterized in that After receiving the first instruction, the first device controls the cursor to move out of the screen of the second device and enter the screen of the third device for display, including: After receiving the first instruction, the first device establishes a connection with the third device; After the first device establishes a connection with the third device, the first device controls the cursor to move out of the screen of the second device and to be displayed on the screen of the third device.

11. The method according to claim 10, characterized in that After the first device establishes a connection with the third device, the method further includes: The second device updates the crossing status of the second device to not crossing; The third device updates the crossing state of the third device to crossing.

12. The method according to claim 10, characterized in that After the first device establishes a connection with the third device, the method further includes: If the first device is disconnected from the third device, the first device controls the cursor to be displayed at a first preset position on the screen of the first device; The first device maintains a communication connection with the second device.

13. The method according to claim 10, characterized in that After the first device establishes a connection with the third device, the method further includes: If the first device is disconnected from the third device, the first device stops intercepting the input event corresponding to the third operation; The first device executes the control instruction corresponding to the input event.

14. The method according to claim 3, characterized in that After the first device determines, in response to the first operation, whether the input device is connected to the first device, the method further includes: If the input device is not connected to the first device, the first device updates the crossing state of the first device to not crossing.

15. The method according to claim 14, characterized in that After the first device determines, in response to the first operation, whether the input device is connected to the first device, the method further includes: If the input device is not connected to the first device, the first device sends a second instruction to the second device; After receiving the second instruction, the second device determines whether the input device is connected to the second device; If the input device is connected to the second device, the second device controls the cursor to move out of the screen of the first device and display it at a second preset position on the screen of the second device.

16. The method according to claim 15, characterized in that After determining whether the input device is connected to the second device, the method further includes: If the input device is connected to the second device, the second device detects a fifth operation, and the fifth operation is used to input the control instruction; The second device stops intercepting the second input event corresponding to the fifth operation; The second device executes the control instruction corresponding to the second input event.

17. The method according to claim 15, characterized in that After determining whether the input device is connected to the second device, the method further includes: If the input device is connected to the second device, the second device updates the crossing state of the second device to not crossing.

18. The method according to claim 15, characterized in that After determining whether the input device is connected to the second device, the method further includes: If the input device is connected to the second device, in response to the sixth operation, the second device determines a first crossing position on the screen of the second device, and the sixth operation is used to move a cursor displayed on the screen of the second device in the second direction; The second device sends the first crossing position to the third device; The third device determines a second crossing position on the screen of the third device according to the first crossing position; The second device controls the cursor to move out of the screen of the second device from the first crossing position; The third device controls the cursor to enter the screen of the third device from the second traversal position.

19. The method according to claim 18, characterized in that After responding to the sixth operation, the method further includes: The second device updates the crossing status of the second device to crossing.

20. The method of claim 18, wherein: After responding to the sixth operation, the method further includes: The second device detects a seventh operation, where the seventh operation is used to input the control instruction; The second device intercepts a third input event corresponding to the seventh operation; The second device sends the third input event to the third device; After receiving the third input event, the third device executes the control instruction corresponding to the third input event.

21. The method of claim 15, wherein: After determining whether the input device is connected to the second device, the method further includes: If the input device is not connected to the second device, the second device determines a fourth device connected to the input device; The second device sends a third instruction to the fourth device; After receiving the third instruction, the fourth device controls the cursor to move out of the screen of the first device and to be displayed on the screen of the second device.

22. The method according to any one of claims 1 to 21, characterized in that In response to the first operation, the first device controls the cursor to move out of the screen of the first device and enter the screen of the second device for display, including: In response to the first operation, the first device determines whether a cursor on a screen of the first device reaches an edge of the screen; If the cursor on the screen of the first device reaches the edge of the screen, the first device controls the cursor to move out of the screen of the first device and to be displayed on the screen of the second device.

23. A communication system, characterized in that: The communication system includes a first device, a second device and a third device; The second device is connected to the first device and the third device respectively; The first device, the second device and the third device in the communication system are configured to execute the method according to any one of claims 1 to 22.

24. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 22 is implemented.

Images