Keyboard and mouse crossing method, communication system and computer readable storage medium

Abstract

The invention relates to the field of terminal control, in particular to a keyboard and mouse crossing method, a communication system and a computer readable storage medium. The method comprises the steps that under the condition that a first cursor of a first input device is displayed on a screen of a first device, the first device monitors a first operation acting on a second input device, the first device is connected with the second input device, the first device is not connected with the first input device, and the first cursor acts on the second input device. The first input device and the second input device are used for inputting a control instruction, and the first operation is used for controlling a cursor action displayed on a screen of the first device; in response to the first operation, the first device controls the first cursor to move out of a screen of the first device, and controls a second cursor of the second input device to be displayed on the screen of the first device. By means of the method, the situation of keyboard and mouse operation conflicts can be effectively reduced, and the user experience can be improved.

Description

Technical Field

[0001] The present application relates to the field of terminal control, and in particular to a keyboard and mouse traversal method, a communication system and a computer-readable storage medium. Background Art

[0002] With the development of terminal technology, the types and number of terminal devices are increasing. At present, one terminal device can no longer meet the needs of users. Users can use multiple terminal devices to meet the needs of work and study. When multiple terminal devices have business that needs to be handled by users, users can use the same set of keyboard and mouse devices to operate different terminals to achieve seamless access between different terminal devices. However, during the application process, the operation conflict between the traversing keyboard and mouse and the local keyboard and mouse is likely to occur, thus affecting the terminal operation and not conducive to the user experience. Summary of the invention

[0003] The present application provides a keyboard and mouse traversal method, a communication system and a computer-readable storage medium, which can effectively reduce the conflict between keyboard and mouse operations and help improve user experience.

[0004] In order to achieve the above objectives, this application adopts the following technical solutions:

[0005] In a first aspect, a keyboard and mouse method is provided, the method comprising:

[0006] In a case where a first cursor of a first input device is displayed on a screen of a first device, the first device detects a first operation acting on a second input device, wherein the first device is connected to the second input device, the first device is not connected to the first input device, the first input device and the second input device are used to input control instructions, and the first operation is used to control a cursor movement displayed on the screen of the first device;

[0007] In response to the first operation, the first device controls the first cursor to move out of the screen of the first device, and controls the second cursor of the second input device to be displayed on the screen of the first device.

[0008] In the embodiment of the present application, the input device may include a mouse, a keyboard, a handwriting board, etc. For example, when the input device is a mouse, the cursor of the input device is a mouse pointer (or mouse cursor).

[0009] In the embodiment of the present application, the first operation may be operations such as dragging, sliding or clicking the mouse.

[0010] In the embodiment of the present application, when the keyboard and mouse are transferred to the first device and the user operates the mouse of the first device, the first device forcibly takes back the keyboard and mouse through the above method. In other words, as long as the local mouse is moved, the keyboard and mouse transfer is stopped. This can avoid the occurrence of transfer confusion.

[0011] In an implementation of the first aspect, the method further includes:

[0012] The first device detects a second operation, where the second operation is used to control a cursor movement displayed on a screen of the first device;

[0013] In response to the second operation, the first device determines whether the input device corresponding to the second operation is the second input device;

[0014] If the input device corresponding to the second operation is the second input device, the first device detects the first operation.

[0015] In the embodiment of the present application, if the input device corresponding to the second operation is not the second input device, the first device continues to detect the second operation. In this case, it means that the second operation is not an operation of the local keyboard and mouse, and the keyboard and mouse do not need to be forcibly recaptured.

[0016] In an implementation of the first aspect, the first device controls the first cursor to move out of a screen of the first device, including:

[0017] The first device detects a second device to which the input device corresponding to the first operation belongs;

[0018] The first device sends a first instruction to the second device to instruct the second device to restore a cursor state;

[0019] After receiving the first instruction, the second device controls the first cursor to move out of the screen of the first device and to be displayed on the screen of the second device.

[0020] In the embodiment of the present application, the first instruction may be referred to as a recovery instruction. The first instruction is a recovery instruction sent by the Android device to the PC when the keyboard and mouse of the PC are passed to the Android device; or a recovery instruction sent by the PC to the Android device when the keyboard and mouse of the Android device are passed to the PC.

[0021] For example, the first device may be a device running an Android system, and the second device may be a device running a Windows system; or the first device may be a device running a Windows system, and the second device may be a device running an Android system. This scenario indicates that the keyboard and mouse of the second device are transferred to the first device. When the first device forcibly takes back the keyboard and mouse, the keyboard and mouse transferred to the first device are transferred back to the second device.

[0022] In an implementation of the first aspect, the method further includes:

[0023] After receiving the first instruction, the second device replies with a first message to the first device to instruct the first device to restore the cursor state;

[0024] After receiving the first information, the first device controls a second cursor of the second input device to be displayed on the screen of the first device.

[0025] The first device controls the second cursor of the second input device to be displayed on the screen of the first device, which is equivalent to the first device restoring the position of its own keyboard and mouse.

[0026] In one implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the first device may be restored based on the positions of the keyboard and mouse on the screen of the first device when the keyboard and mouse are passed from the second device.

[0027] In another implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the first device can be restored according to the crossing position of the first cursor on the screen of the first device when the first device sends the first instruction.

[0028] In another implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the first device may be restored to a preset position, for example, the preset position is the center of the screen of the first device.

[0029] In the embodiment of the present application, the first instruction and the first information are used to enable the first device and the second device to mutually confirm the restoration of the cursor state, thereby reducing the probability of misjudgment.

[0030] In an implementation of the first aspect, the method further includes:

[0031] After the second device receives the first instruction, the second device stops intercepting the first input event corresponding to the third operation, wherein the third operation is used to control the movement of the first cursor of the first input device;

[0032] In response to the third operation, the second device executes the control instruction corresponding to the first input event.

[0033] In one implementation, when the first device is an Android device, the keyboard and mouse service module in the first device can call a preset interface of the input framework to intercept the first input event corresponding to the third operation. By intercepting the event, the mouse passes from the Android device to the PC, and the input framework of the Android system intercepts the monitored keyboard and mouse events, so that the service on the Android device does not respond to the keyboard and mouse events, so that the Android device does not affect the mouse events.

[0034] In one implementation, when the first device is a device deploying a Windows system, the first device can intercept the first input event corresponding to the third operation through an interception event. The interception event is used to intercept keyboard and mouse events. The indication information of the interception event is used to instruct the input framework to set a keyboard hook, a mouse hook, and a shortcut hook. The hook is a platform of the Windows message processing mechanism, on which an application can set a subroutine to monitor a certain message of a specified window, and the monitored window can be created by other processes. When the message arrives, it is processed before the target window processing function. The hook mechanism allows an application to intercept and process window messages or specific events. The hook is actually a program segment for processing messages, which is hung into the system through a system call. Whenever a specific message is sent, before it reaches the destination window, the hook program first captures the message, that is, the hook function first obtains control. At this time, the hook function can process (change) the message, or continue to pass the message without processing, or force the message to end. The keyboard hook is used to obtain the user's input operations through the keyboard. For example, the keyboard hook is used to obtain the user's operation of inputting "memo information" through the keyboard. The mouse hook is used to obtain the user's control operations through the mouse. For example, the mouse hook is used to obtain the user's click or drag operations through the mouse. The shortcut key hook is used to obtain the user's shortcut key operations. For example, the shortcut key hook is used to obtain the user's Ctrl+C, Ctrl+V, ATL+TAB, or Ctrl+Alt+DELETE operations. By intercepting events, the mouse passes from the PC to the Android device, and the Windows system input framework intercepts the monitored keyboard and mouse events, so that the business on the PC does not respond to keyboard and mouse events, so that the PC does not affect the mouse events.

[0035] In the embodiment of the present application, the second device stops intercepting input events corresponding to keyboard and mouse operations and executes control instructions corresponding to the operations of the input device, so that the input device can be flexibly switched between different devices, avoiding cumbersome operations and improving user experience. In addition, by stopping intercepting input events, the second device can respond to keyboard and mouse operations in a timely manner, thereby ensuring the reliability of keyboard and mouse crossing.

[0036] In an implementation of the first aspect, the method further includes:

[0037] After the second device receives the first instruction, the second device updates the crossing status of the second device to not crossing.

[0038] In the embodiment of the present application, by timely updating the crossing status of the second device, the occurrence of crossing confusion can be effectively reduced, which is beneficial to improving the reliability of keyboard and mouse crossing.

[0039] In an implementation of the first aspect, the method further includes:

[0040] After the first device receives the first information, the first device stops intercepting a second input event corresponding to a fourth operation, wherein the fourth operation is used to control the movement of a second cursor of the second input device;

[0041] In response to the fourth operation, the first device executes the control instruction corresponding to the second input event.

[0042] In the embodiment of the present application, the first device stops intercepting input events corresponding to keyboard and mouse operations and executes control instructions corresponding to the operations of the input device, so that the input device can be flexibly switched between different devices, avoiding cumbersome operations and improving user experience. In addition, by stopping intercepting input events, the first device is guaranteed to respond to keyboard and mouse operations in a timely manner, thereby ensuring the reliability of keyboard and mouse crossing.

[0043] In an implementation of the first aspect, the method further includes:

[0044] After the first device receives the first information, the first device updates the crossing status of the first device to not crossing.

[0045] In the embodiment of the present application, by timely updating the crossing status of the first device, the occurrence of crossing confusion can be effectively reduced, which is beneficial to improving the reliability of keyboard and mouse crossing.

[0046] In an implementation of the first aspect, after the first device detects a second device to which the input device corresponding to the first operation belongs, the method further includes:

[0047] The first device acquires device types of the first device and the second device, wherein the device types include a first type and a second type, and the first type of device is communicatively connected with each of the second type of devices respectively;

[0048] If the device type of the first device is the first type and the device type of the second device is the second type, or the device type of the first device is the second type and the device type of the second device is the first type, the first device sends the first instruction to the second device to instruct the second device to restore the cursor state.

[0049] For example, the first device is a PC, and the second device is an Android device. In an application scenario where the PC is the communication center, a local keyboard and mouse are lost on the PC side, and the PC performs a forced recapture. In this scenario, the PC passes the keyboard and mouse of the Android device back to the Android device, and the PC screen displays the mouse pointer of its own keyboard and mouse.

[0050] In an implementation manner of the first aspect, after the first device acquires the device types of the first device and the second device, the method further includes:

[0051] If the device types of the first device and the second device are both the second type, the first device sends a second instruction to the second device to instruct the second device to restore the cursor state;

[0052] The first device sends a third instruction to a third device to instruct the third device to update a traversal state, wherein the device type of the third device is the first type.

[0053] In the embodiment of the present application, the second instruction may be referred to as a recovery instruction. The second instruction is a recovery instruction sent by the Android device at the receiving end to the Android device at the source end when the keyboard and mouse of the Android device are passed to the Android device. The third instruction may also be referred to as a recovery instruction, and the third instruction is a recovery instruction sent by the Android device at the receiving end to the PC when the keyboard and mouse of the Android device are passed to the Android device.

[0054] For example, the first device is an Android device, and the second device is an Android device. In an application scenario with a PC as the communication center, a local keyboard and mouse error occurs on the Android device side, and the Android device performs a forced recapture. In this scenario, the first device not only needs to send a command to restore the cursor state to the second device, but also needs to inform the PC to stop the keyboard and mouse crossing, so that the crossing state of each terminal in the communication system remains consistent, thereby reducing the occurrence of keyboard and mouse crossing confusion.

[0055] In an implementation manner of the first aspect, after the first device sends the second instruction to the third device, the method further includes:

[0056] The third device receives the third instruction and updates the crossing status of the third device to not crossing.

[0057] In the embodiment of the present application, by timely updating the crossing status of the first device, the occurrence of crossing confusion can be effectively reduced, which is beneficial to improving the reliability of keyboard and mouse crossing.

[0058] In an implementation manner of the first aspect, the first type is a Windows system, and the second type is an Android system.

[0059] In an implementation of the first aspect, the first device detecting a second device to which an input device corresponding to the first operation belongs includes:

[0060] The first device acquires device information of an input device corresponding to the first operation;

[0061] The first device determines, according to the device information, a second device to which the input device corresponding to the first operation belongs.

[0062] Exemplarily, the device information may be a device ID or a physical address of the device, etc.

[0063] In a second aspect, a communication system is provided, the communication system comprising a first device, a second device and a third device;

[0064] The first device, the second device and the third device in the communication system are used to execute the method as described in any one of the first aspects.

[0065] According to a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of the first aspects is implemented.

[0066] According to a fourth aspect, a computer program product is provided, wherein the computer program product includes a computer program code, and when the computer program code is executed on a computer, the computer implements the method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a schematic diagram of a communication system provided in an embodiment of the present application;

[0068] Figure 2 is a schematic diagram of a software architecture of a Windows system provided in an embodiment of the present application;

[0069] Figure 3 is a schematic diagram of a software architecture of an Android system provided in an embodiment of the present application;

[0070] Figure 4 is an interactive schematic diagram of establishing a keyboard and mouse connection provided in an embodiment of the present application;

[0071] Figure 5 This is a schematic diagram of an interface of a computer manager provided in an embodiment of the present application;

[0072] Figure 6 It is a position diagram provided in an embodiment of the present application;

[0073] Figure 7 is a position diagram provided by another embodiment of the present application;

[0074] Figure 8 is a schematic diagram of a keyboard and mouse traversal provided in an embodiment of the present application;

[0075] Fig. 9 is a schematic diagram of a keyboard and mouse traversal scenario provided in an embodiment of the present application;

[0076] Fig.10 is a schematic diagram of an interaction scenario provided in an embodiment of the present application;

[0077] Fig.11 It is a schematic diagram of the implementation principle of a traversal method provided in an embodiment of the present application;

[0078] Fig.12 It is a schematic diagram of the interaction process of a keyboard and mouse traversal method provided in an embodiment of the present application;

[0079] Fig.13 is a schematic diagram of a screen crossing position provided in an embodiment of the present application;

[0080] Fig.14 is a schematic diagram of another keyboard and mouse traversal scenario provided in an embodiment of the present application;

[0081] Fig.15 It is a schematic diagram of the implementation principle of a traversal method provided in an embodiment of the present application;

[0082] Fig.16 It is a schematic diagram of the interaction process of a keyboard and mouse traversal method provided in an embodiment of the present application;

[0083] Fig.17 is a schematic diagram of a keyboard and mouse traversal scenario provided in an embodiment of the present application;

[0084] Fig.18 is a schematic diagram of a keyboard and mouse traversal scenario provided in an embodiment of the present application;

[0085] Fig.19 It is a schematic diagram of the interaction process of a keyboard and mouse traversal method provided in an embodiment of the present application;

[0086] Fig. 20 It is a schematic diagram of the interaction process of a keyboard and mouse traversal method provided in an embodiment of the present application;

[0087] Fig.21 This is a schematic diagram of a keyboard and mouse crossing conflict scenario provided by an embodiment of the present application;

[0088] Fig. 22 It is a schematic diagram of the interaction process of a keyboard and mouse traversal method provided in an embodiment of the present application;

[0089] Fig.23 It is a schematic diagram of the interaction process of a keyboard and mouse traversal method provided in an embodiment of the present application;

[0090] Fig.24 It is a schematic diagram of a keyboard and mouse crossing forced recapture scenario provided in an embodiment of the present application. DETAILED DESCRIPTION

[0091] In the following description, specific details such as specific system structures and technologies are provided for illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0092] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0093] It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more than two; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0094] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", "fourth", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0095] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0096] With the development of terminal technology, the types and number of terminal devices are increasing. At present, one terminal device can no longer meet the needs of users. Users can use multiple terminal devices to meet the needs of work and study. However, when multiple terminal devices have services that need to be handled by users, users need to switch between different terminal devices. The operation is cumbersome and not conducive to user experience.

[0097] The embodiment of the present application provides a keyboard and mouse traversal method, which can operate different terminal devices through a set of input devices to achieve seamless access between different terminal devices.

[0098] The method of the embodiment of the present application can be applied to a terminal device with a screen and an input device. For example, the terminal device in the embodiment of the present application can be a personal computer (PC) (such as a desktop computer, a laptop, a small laptop or an ultrabook, etc.), a tablet computer (referred to as a tablet) and a mobile phone. The embodiment of the present application does not specifically limit the specific form of the terminal device. For the sake of convenience, the following embodiments are introduced by taking PC (or computer), tablet and mobile phone as examples.

[0099] In the embodiment of the present application, the input device may be a keyboard, a mouse, a handwriting board (which may include a stylus), or other devices used to input information to the terminal device. The embodiment of the present application does not specifically limit the specific form of the input device. For ease of explanation, the keyboard and the mouse are referred to as keyboard and mouse in the following embodiments, and the input device is a keyboard and mouse as an example for introduction.

[0100] In the embodiment of the present application, keyboard and mouse traversal can be implemented between two or more terminal devices. The following is an introduction using three terminal devices as an example.

[0101] See also Figure 1 , is a schematic diagram of a communication system provided in an embodiment of the present application. Figure 1 As shown, the communication system includes a computer 11, a mobile phone 12 and a tablet 13. The computer 11 is connected to the mobile phone 12 and the tablet 13 for communication.

[0102] Figure 1 In the communication system shown, keyboard and mouse can be passed between the computer 11 and the mobile phone 12, keyboard and mouse can be passed between the computer 11 and the tablet 13, and keyboard and mouse can be passed between the mobile phone 12 and the tablet 13 through the computer 11. In other words, the computer 11, the mobile phone 12 and the tablet 13 can share the mouse and keyboard.

[0103] It is understandable that the computer 11 can also achieve keyboard and mouse crossover only with the mobile phone 12 or the tablet 13. In other words, the computer 11 and the mobile phone 12 share the mouse and keyboard, or the computer 11 and the tablet 13 share the mouse and keyboard.

[0104] In the embodiment of the present application, the keyboard and mouse of the computer 11 may be a keyboard and mouse connected to the computer 11 , and the keyboard and mouse of the mobile phone 12 or tablet 13 may be a keyboard and mouse external to the mobile phone 12 or tablet 13 .

[0105] The left or right side of the computer 11 can realize keyboard and mouse crossover with a device. For example, the computer 11 can support the mouse to cross from the left side of the computer 11 screen to the tablet 13, and can also support the mouse to cross from the right side of the computer 11 screen to the mobile phone 12. The computer 11 can realize keyboard and mouse crossover with one device (mobile phone 12 or tablet 13) at a time. In other words, if a device is currently in the crossover state, it will no longer cross over with other devices.

[0106] In the embodiment of the present application, the software system of the terminal device involved in keyboard and mouse traversal may be a Windows system or an Android system. Figure 1 The operating system of the computer 11 shown in the figure is Windows system, and the operating systems of the mobile phone 12 and the tablet 13 are Android system. Among them, the device deploying the Android system can be called an Android device.

[0107] In order to better understand the embodiments of the present application, the software architecture of the Windows system and the Android system of the embodiments of the present application are introduced below.

[0108] For example, see Figure 2 , is a schematic diagram of a software architecture of a Windows system provided in an embodiment of the present application. Figure 2 As shown, the software architecture of the Windows system may include a driver layer, a basic framework layer, and an application layer.

[0109] The driver layer includes a virtual driver module and an input driver module. The virtual driver module is used to set a virtual driver for the input device of the Android device. The input driver module is used to provide a human interface device (HID) driver to drive the input devices (such as keyboard and mouse) connected to the PC.

[0110] The basic framework layer includes an input framework module, which is used to report input events to each business module in the application layer so that each business module in the application layer can implement corresponding functions. For example, the input framework module reports keyboard and mouse events to the keyboard and mouse business module in the application layer so that the keyboard and mouse business module can process the keyboard and mouse events, thereby realizing keyboard and mouse crossing.

[0111] The application layer includes a communication module (such as Magic Link) and a keyboard and mouse service module. The communication module is used to provide a connection channel and data transmission capability between devices. The keyboard and mouse service module is used to provide a keyboard and mouse crossing function. Among them, the keyboard and mouse service module includes a connection scheduling submodule, a keyboard and mouse management submodule and a virtual driver loading submodule. The connection scheduling submodule is used to provide a connection channel and data transmission capability with the communication module. The keyboard and mouse management submodule is used to call the connection scheduling submodule and the virtual driver loading submodule, and provide the ability to process keyboard and mouse events. The virtual driver loading submodule is used to inject the keyboard and mouse events processed by the keyboard and mouse management submodule into the virtual driver module in the driver layer, so as to report the keyboard and mouse events to the input framework module in the system base layer through the virtual driver module.

[0112] It should be noted that the communication module is a module for communication between different devices. The data obtained by other modules must be transmitted to other devices first, and then transmitted to other devices by the communication module. In subsequent embodiments, in order to simplify the steps, in addition to communication module communication between different devices, other modules perform cross-device communication. It can be understood that other modules first transmit the data to the communication module, and then the communication module transmits it to the communication module of other devices, and then the communication module of other devices transmits it to the module of the required data.

[0113] It should be noted that the data interaction between the keyboard and mouse service module and the communication module needs to go through the connection scheduling submodule. For example, the data packet of the keyboard and mouse event sent by other devices is first transmitted to the communication module, and then sent by the communication module to the connection scheduling subunit, and then sent by the connection scheduling subunit to the keyboard and mouse management submodule. In subsequent embodiments, in order to simplify the steps, there are situations where other submodules in the keyboard and mouse service module other than the connection scheduling submodule communicate with the communication module, which can be understood as other submodules first transmitting data to the connection scheduling submodule, and then the connection scheduling submodule transmitting it to the communication module, or the communication module first transmitting data to the connection scheduling submodule, and then the connection scheduling submodule transmitting it to other submodules.

[0114] For example, see Figure 3 , is a schematic diagram of a software architecture of an Android system provided in an embodiment of the present application. Figure 3 As shown, the software architecture of the Android system may include a driver layer, a basic framework layer, and an application layer.

[0115] The driver layer includes a virtual input driver module (Uinput Driver), wherein the virtual input driver module is used to drive the input device of the Android device.

[0116] The basic framework layer includes an input framework module, which is used to report input events to each business module in the application layer so that each business module in the application layer can implement corresponding functions. For example, the input framework module reports keyboard and mouse events to the keyboard and mouse business module in the application layer so that the keyboard and mouse business module can process the keyboard and mouse events, thereby realizing keyboard and mouse crossing.

[0117] The application layer includes a communication module (such as Magic Link) and a keyboard and mouse service module. The communication module is used to provide a connection channel and data transmission capability between devices. The keyboard and mouse service module is used to provide a keyboard and mouse crossing function. Among them, the keyboard and mouse service module includes a connection scheduling submodule and a keyboard and mouse management submodule. The connection scheduling submodule is used to provide a connection channel and data transmission capability with the communication module. The keyboard and mouse management submodule is used to call the connection scheduling submodule, provide the ability to process keyboard and mouse events, and inject the processed keyboard and mouse events into the virtual input driver module in the driver layer, so as to report the keyboard and mouse events to the input framework module in the system base layer through the virtual input driver module.

[0118] It should be noted that the communication module is a module for communication between different devices. The data obtained by other modules must be transmitted to other devices first, and then transmitted to other devices by the communication module. In subsequent embodiments, in order to simplify the steps, in addition to communication module communication between different devices, other modules perform cross-device communication. It can be understood that other modules first transmit the data to the communication module, and then the communication module transmits it to the communication module of other devices, and then the communication module of other devices transmits it to the module of the required data.

[0119] It should be noted that the data interaction between the keyboard and mouse service module and the communication module needs to go through the connection scheduling submodule. For example, the data packet of the keyboard and mouse event sent by other devices is first transmitted to the communication module, and then sent by the communication module to the connection scheduling subunit, and then sent by the connection scheduling subunit to the keyboard and mouse management submodule. In subsequent embodiments, in order to simplify the steps, there are situations where other submodules in the keyboard and mouse service module other than the connection scheduling submodule communicate with the communication module, which can be understood as other submodules first transmitting data to the connection scheduling submodule, and then the connection scheduling submodule transmitting it to the communication module, or the communication module first transmitting data to the connection scheduling submodule, and then the connection scheduling submodule transmitting it to other submodules.

[0120] In order to better understand the embodiments of the present application, the embodiments of the present application are combined with the above Figure 1 The communication system shown and Figure 2 , Figure 3 The software architecture shown provides a detailed introduction to the method provided in the embodiment of the present application.

[0121] In an embodiment of the present application, the condition for realizing keyboard and mouse crossing between terminal devices may be that a communication connection has been established between the terminal devices. For example, the PC and the Android device may be connected via WiFi or Bluetooth communication, wherein WiFi may include a local area network or peer to pee (P2P); or may be connected via a wired communication, such as via a data cable such as type-A, type-B or type-C. In other implementations, the PC and the Android device log in using the same account. For example, when the computer 11, the mobile phone 12 and the tablet 13 log in using the same account, the computer 11, the mobile phone 12 and the tablet 13 establish a communication connection.

[0122] For example, see Figure 4 , is an interactive schematic diagram of establishing a keyboard and mouse connection provided in an embodiment of the present application. Figure 4 As shown, the process of establishing a keyboard and mouse connection between a PC and any Android device may include the following steps:

[0123] S401, the keyboard and mouse management submodule of the PC is started.

[0124] The keyboard and mouse management submodule in the keyboard and mouse service module of the PC can be automatically started when the PC is turned on, or can be manually started by the user. For example, in the case of manual start by the user, the user can turn on the keyboard and mouse pass-through function through the setting interface on the PC. In response to the user's operation of turning on the keyboard and mouse pass-through function, the keyboard and mouse service module of the PC is started, and correspondingly, the keyboard and mouse management submodule in the keyboard and mouse service module of the PC is started.

[0125] S402, the keyboard and mouse management submodule of the PC sends instruction information a to the connection scheduling submodule of the PC.

[0126] The indication information a is used to instruct the connection scheduling submodule of the PC to establish a communication connection with the Android device.

[0127] S403, after receiving the indication information a, the connection scheduling submodule of the PC sends a request information b to the communication module of the PC.

[0128] The request information b is used to request the communication module of the PC to inform the detected online Android device.

[0129] The communication module of the PC can detect whether there is an online Android device. If there is an online Android device, based on the request information b, the communication module of the PC sends the device identification (identity, ID) of the online Android device to the keyboard and mouse management submodule of the PC through the connection scheduling submodule of the PC. If there is no online Android device, the communication module of the PC can monitor in real time whether there is an online Android device, or detect whether there is an online Android device at every preset period.

[0130] The connection scheduling submodule of the PC sends a request message b to the communication module of the PC. It can also be understood that the connection scheduling submodule of the PC registers the device online and offline fences in the communication module of the PC. When the communication module of the PC detects the existence of an online device, the device ID is returned to the keyboard and mouse management submodule of the PC through the connection scheduling submodule.

[0131] S404, the keyboard and mouse management submodule of the Android device is started.

[0132] The keyboard and mouse management submodule in the keyboard and mouse service module of the Android device can be automatically started when the Android device is turned on, or can be manually started by the user. For example, in the case of manual start by the user, the user can turn on the keyboard and mouse pass-through function through the setting interface on the Android device. In response to the user's operation of turning on the keyboard and mouse pass-through function, the keyboard and mouse service module of the Android device is started, and correspondingly, the keyboard and mouse management submodule in the keyboard and mouse service module of the Android device is started.

[0133] S405, after receiving the instruction information b, the communication module of the PC discovers each other with the communication module of the Android device to form a trust loop.

[0134] As mentioned above, the premise of keyboard and mouse traversal between PC and Android device is that the communication connection has been established between PC and Android device. The communication connection here means that the communication module of PC and the communication module of Android device can discover each other and establish a trust loop.

[0135] It should be noted that the above S402 is a step that will be executed after the keyboard and mouse management submodule of the PC is started, and the embodiment of the present application does not limit the order of execution of the two steps S402 and S404. In other words, if before S402, the step S404 has been executed, that is, the keyboard and mouse management submodule of the Android device has been started, then after executing S402, the communication module of the PC can detect that the Android device is online, so that S405 can be executed. If before S402, the step S404 is not executed, that is, the keyboard and mouse management submodule of the Android device is not started, then after executing S402, the communication module of the PC detects in real time or at every preset period whether there is an online Android device, until the keyboard and mouse management submodule of the Android device is started, and the communication module of the PC detects that there is an online Android device, and then executes S405.

[0136] A trust ring is used to indicate that a communication connection has been established between different terminal devices. For example, if a PC and an Android device are in the same Bluetooth range, or connected to the same WiFi, or connected via the same data cable, or logged into the same account, the communication module of the PC and the communication module of the Android device can sense and discover each other, forming a trust ring, and the communication module of the PC can obtain the device ID of the Android device.

[0137] The device ID can be at least one of numbers, letters or symbols, which is not limited in this embodiment of the present application.

[0138] The online Android device may be one or more, which is not limited in the present embodiment. If there are multiple online Android devices, the communication module of the PC may send the device IDs of the multiple Android devices to the keyboard and mouse service module of the PC.

[0139] In one implementation, the keyboard and mouse service module of the PC may include a device management submodule. If the PC is communicating with the Android device for the first time, the device management submodule of the PC may authenticate the Android device based on the device ID. After the authentication is passed, the connection scheduling submodule of the PC notifies the communication module of the PC to execute S405 and store the device ID. During the next communication connection process, when the communication module of the PC detects the online Android device, the device ID of the Android device is reported to the device management submodule of the PC through the connection scheduling submodule of the PC. The device management submodule of the PC searches whether the device ID has been stored. If the device ID has been stored, the connection scheduling submodule of the PC sends an indication message to the communication module of the PC to instruct the communication module of the PC to execute S405.

[0140] S406, after the trust ring is formed, the communication module of the Android device sends a notification c to the connection scheduling submodule of the Android device.

[0141] Notification c is used to instruct the connection scheduling submodule of the Android device to report to the keyboard and mouse management submodule of the Android device that the communication connection with the PC is successful.

[0142] S407, after receiving the notification c, the connection scheduling submodule of the Android device sends a notification d to the keyboard and mouse management submodule of the Android device.

[0143] Notification d is used to indicate that the communication connection with the PC is successful.

[0144] S408, after the trust ring is formed, the communication module of the PC sends a notification e to the connection scheduling submodule of the PC.

[0145] Notification e is used to instruct the connection scheduling submodule of the PC to report to the keyboard and mouse management submodule of the PC that the communication connection with the Android device is successful.

[0146] S409, after receiving the notification e, the connection scheduling submodule of the PC sends a notification f to the keyboard and mouse management submodule of the PC.

[0147] Notification f is used to indicate that the communication connection with the Android device is successful.

[0148] It should be noted that steps S406 and S408 are not performed in any particular order and can be processed in parallel.

[0149] At this point, the keyboard and mouse connection between the PC and the Android device is successful.

[0150] It is understandable that in order to achieve keyboard and mouse traversal between PC and Android device, it is necessary not only to establish a communication connection between PC and Android device (i.e., a trust loop is formed between the communication module of PC and the communication module of Android device), but also to establish a keyboard and mouse connection between PC and Android device (i.e., an event connection between the keyboard and mouse service module of PC and the keyboard and mouse service module of Android device). In other words, Figure 4 The figure shows the process of establishing a keyboard and mouse connection between a PC and an Android device. The keyboard and mouse connection between the PC and the Android device is successful, indicating that the event connection between the keyboard and mouse service module of the PC and the keyboard and mouse service module of the Android device is successful.

[0151] It should be noted that Figure 4 The embodiment shows the process of establishing a keyboard and mouse connection with a PC as the center, in other words, the keyboard and mouse connection is initiated by the PC. In the application scenario centered on the PC, if the keyboard and mouse crossing between the PC and multiple Android devices is realized, it is necessary to establish a keyboard and mouse connection between the PC and each Android device respectively. In other embodiments, the keyboard and mouse connection can also be established with the Android device as the center, that is, the keyboard and mouse connection is initiated by the Android device. In the application scenario centered on the Android device, if the keyboard and mouse crossing between the Android device and multiple terminal devices is realized, it is necessary to establish a keyboard and mouse connection between the Android device and each terminal device respectively. For the sake of convenience of explanation, the application scenarios involved in the embodiments of the present application are all PC-centric scenarios.

[0152] There are two ways to establish a keyboard and mouse connection between a PC and an Android device. One possible implementation is to automatically establish a keyboard and mouse connection. In this way, after the keyboard and mouse management submodule of the PC is started in S401, the subsequent steps are automatically executed, and the PC can independently select a connectable device. Another possible implementation is to manually establish a keyboard and mouse connection. In this way, after the keyboard and mouse management submodule of the PC is started in S401, when the communication module of the PC detects an online Android device, the user can manually select a connectable device in the visual interface. After the user selects, the steps after S401 are executed.

[0153] The setting options for the keyboard and mouse connection mode may be located in different locations in different devices. For example, the setting options for the keyboard and mouse connection mode may be in the computer manager of the computer 11. The setting options for the keyboard and mouse connection mode may be in the control center or settings of the mobile phone 12 or the tablet 13.

[0154] For example, see Figure 5 , is a schematic diagram of a computer manager interface provided by an embodiment of the present application. Figure 5As shown, the computer manager interface may include options such as home page, multi-screen collaboration, smart interconnection, online skills, official services, smart audition, and system optimization. When the computer 11 detects that the user clicks the smart interconnection option with the mouse, the keyboard and mouse sharing interface can be displayed. The keyboard and mouse sharing interface includes two options. Among them, the first option is "touch the mouse pointer twice on the side edge of the computer screen to establish a connection." The second option is "automatically adjust the screen arrangement as the device placement changes."

[0155] It can be understood that the first option is the above-mentioned option of automatically establishing a keyboard and mouse connection. When this option is selected, the computer 11 can automatically connect to a device that meets the keyboard and mouse cross-condition. When the first option (i.e., the option of automatically establishing a communication connection) is not selected, the computer 11 provides a method of manually establishing a keyboard and mouse connection to connect to a device that meets the keyboard and mouse cross-condition.

[0156] As an example of manually establishing a keyboard and mouse connection, Figure 5 As shown in , the first option is not selected. At this time, the computer 11 provides a manual connection method. The keyboard and mouse sharing interface can display connected devices and online devices. Figure 5 In the keyboard and mouse sharing interface shown in (a), a mobile phone 12 is connected to the left side of the computer 11, and there are two online devices (devices with established trust loops), including a tablet 13 and a mobile phone 14. The user can select a device from the two online devices by clicking or dragging. Figure 5 As shown in (a) of FIG. 1 , the user operates the mouse to control the mouse pointer to click the icon of the tablet 13. In response to the operation of clicking the icon of the tablet 13, the computer 11 establishes a keyboard and mouse connection with the tablet 13 and displays the following Figure 5 The keyboard and mouse sharing interface shown in (b) of FIG. Figure 5 As shown in (b) , in the keyboard and mouse sharing interface, a tablet 13 is connected to the right side of the computer 11 .

[0157] like Figure 5 In the keyboard and mouse sharing interface shown in (a) and (b) of FIG. 1 , a selection mark 501 may be displayed on the icon of the device connected to the computer 11. The user may disconnect the connection between the PC and the device by operating the mouse to control the mouse pointer to click the selection mark 501 or drag the icon of the connected device. For example, the user may disconnect the connection between the PC and the device by operating the mouse to control the mouse pointer to click the selection mark 501 or drag the icon of the connected device. Figure 5 In the interface shown in (b) of FIG. 1 , the user clicks the selection mark 501 on the mobile phone on the left side of the computer 11. In response to the user operation, the computer 11 disconnects the keyboard and mouse connection between the mobile phone 12 and displays the following Figure 5 The keyboard and mouse sharing interface shown in (c) of FIG. Figure 5As shown in (c), in the keyboard and mouse sharing interface, no device is connected to the left side of the computer 11, a tablet 13 is connected to the right side of the computer 11, and the mobile phone 12 is displayed in the area where the device can be connected.

[0158] In the above-mentioned manual establishment of the keyboard and mouse connection, the PC determines the device's crossing direction according to the position of the device selected by the user in the keyboard and mouse sharing interface. Figure 5 As shown in (b), the mobile phone 12 is on the left side of the computer 11, and the tablet 13 is on the right side. The computer 11 determines that the crossing direction of the mobile phone 12 is the left side (that is, when the mouse in the computer 11 is slid to the left, the mouse pointer can reach the screen of the mobile phone 12), and the crossing direction of the tablet 13 is the right side (that is, when the mouse in the computer 11 is slid to the right, the mouse pointer can reach the screen of the tablet 13).

[0159] As an example of automatically establishing a keyboard and mouse connection, if Figure 5 The first option in is selected. When the computer 11 detects that the mouse pointer touches the side edge of the computer screen twice in a row, the computer 11 can automatically connect to the device that meets the keyboard and mouse crossing conditions (such as connecting to the same network and / or logging into the same account). For example, after executing the above S403, if the communication module of the computer 11 detects an online Android device, S405 is executed. If the communication module of the computer 11 detects multiple online Android devices, it can establish a keyboard and mouse connection with each online Android device respectively, or automatically select one or several of them to establish a keyboard and mouse connection.

[0160] In the mode of automatically establishing a keyboard and mouse connection, the PC can automatically set the traversal direction of the connected device, or the traversal direction of the connected device can be manually set by the user.

[0161] As an example of automatically setting the traversal direction, when the user turns on the option of automatically identifying the orientation, the PC can detect the position of the device with the established keyboard and mouse connection through ultrasound, and automatically set the traversal direction of the device with the established keyboard and mouse connection based on the detected position.

[0162] For example, in Figure 5 In the interface shown, if the user checks or selects the option "Automatically adjust screen arrangement as device placement changes", the PC detects the user's check or selection operation and, in response to the operation, can automatically identify the location of the currently connected device. Figure 1In the application scenario shown, when the user turns on the option of automatically identifying the orientation, the mobile phone 12 or tablet 13 can detect its own motion through the motion sensor. When the motion stops, the position of the computer 11 relative to the mobile phone 12 or tablet 13 is calculated through the data obtained by the ultrasonic sensor, and the position is transmitted to the computer 11. The computer 11 receives the position and can determine that the mobile phone 12 is on the left side of the computer 11 and the tablet 13 is on the right side of the computer 11; then the computer 11 sets the crossing direction of the mobile phone 12 to the left (that is, when the mouse in the computer 11 slides to the left, the mouse pointer can reach the screen of the mobile phone 12), and the computer 11 sets the crossing direction of the tablet 13 to the right (that is, when the mouse in the computer 11 slides to the right, the mouse pointer can reach the screen of the tablet 13).

[0163] It is understandable that when the user turns on the option of automatic orientation identification, if the position of the device to which the PC currently has a keyboard and mouse connection changes, the PC can detect the position change of the device to which the keyboard and mouse connection has been established through ultrasound and automatically adjust the crossing direction of the connected device.

[0164] It should be noted that the keyboard and mouse can not only move to the left or right side of the screen, but also move to the top or bottom of the screen, which is not limited in the embodiments of the present application.

[0165] In one possible implementation, the screen of the terminal device can be divided into the left side, the right side, and the middle. For example, the computer 11 can establish a coordinate system with the center of the screen as the origin, the parallel lines of the long side of the screen, and the parallel lines of the wide side of the screen, so as to divide the left area, the right area, and the middle area. If the computer 11 detects that the mobile phone 12 and / or the tablet 13 are in the left area, it can be determined that the mobile phone 12 and / or the tablet 13 are on the left side of the computer 11. If the computer 11 detects that the mobile phone 12 and / or the tablet 13 are in the right area, it can be determined that the mobile phone 12 and / or the tablet 13 are on the right side of the computer 11. If the computer 11 detects that the mobile phone 12 and / or the tablet 13 are in the middle area, it can be determined that the mobile phone 12 and / or the tablet 13 are in the middle of the computer 11.

[0166] For example, see Figure 6 , is a position diagram provided in an embodiment of the present application. Figure 6 As shown, the computer 11 sets the area between (-45°, 45°) in the coordinate system as the right area, the area between (45°, 135°) and (-45°, -135°) as the middle area, and the area between (-135°, 135°) as the left area. If the angle of the mobile phone 12 relative to the computer 11 is between (-45°, 45°), the computer 11 detects that the mobile phone 12 is in the right area, and it can be determined that the mobile phone 12 is on the right side of the computer 11.

[0167] In another possible implementation, the screen of the terminal device may be divided into a left side, a right side, an upper side and a lower side. Figure 7 , is a position diagram provided by another embodiment of the present application. Figure 7 As shown, the computer 11 sets the area between (-45°, 45°) in the coordinate system as the right area, the area between (45°, 135°) as the upper area, the area between (-45°, -135°) as the lower area, and the area between (-135°, 135°) as the left area. If the angle of the mobile phone 12 relative to the computer 11 is between (45°, 135°), the computer 11 detects that the mobile phone 12 is in the upper area, and it can be determined that the mobile phone 12 is above the computer 11.

[0168] It should be noted that Figure 6 and Figure 7 The angle range of the area division shown in is only an example. The angle range of each area can be adjusted according to the user experience. For example, the angle range of the middle area can be adjusted from (45°, 135°) to (30°, 150°) and from (-45°, -135°) to (-30°, -150°) according to the user experience.

[0169] As an example of manually setting the traversal direction, when the user manually sets the traversal direction of the connected device, when the PC detects that the mouse pointer is at the edge of the computer screen, a bubble box can be displayed at the edge of the screen to indicate that there is a traversable device (a device with a keyboard and mouse connection). When the user operates the bubble box and selects a traversable device, the PC sets the traversal direction of the device selected by the user according to the edge of the current mouse pointer on the computer screen.

[0170] For example, see Figure 8 , is a schematic diagram of a keyboard and mouse traversal provided in an embodiment of the present application. Figure 8As shown, the traversable devices of the computer 11 include a mobile phone 12 and a tablet 13. When the computer 11 detects that the mouse pointer is at the right edge of the computer screen, a bubble box 801 is displayed at the right edge of the computer screen to indicate the presence of a traversable device. The bubble box 801 displays an icon 8011 of the mobile phone 12 and an icon 8012 of the tablet 13. The user can operate the mouse to control the mouse pointer to click the icon 8011 of the mobile phone 12. In response to the user operation, the computer 11 sets the traversal direction of the mobile phone 12 to the right, that is, the mouse in the computer 11 slides to the right, and the mouse pointer can reach the screen of the mobile phone 12. The user can also operate the mouse to control the mouse pointer to click the icon 8012 of the tablet 13. In response to the user operation, the computer 11 sets the traversal direction of the tablet 13 to the right, that is, the mouse of the computer 11 slides to the right, and the mouse can reach the screen of the tablet 13.

[0171] Among them, the bubble box is only an example of a name, and the embodiments of the present application do not limit this. The number of traversable devices (devices that have formed a trust loop with the PC) can be one or more, and the embodiments of the present application do not limit this. When the number of traversable devices is one, the number of bubble boxes can be one. The bubble box can display the icon and model of the traversable device. For example, if the traversable device is a mobile phone, the mobile phone icon and the model of the mobile phone can be displayed, or the icon and / or model of the traversable device may not be displayed, and the embodiments of the present application do not limit this.

[0172] When there are multiple traversable devices, the number of bubble boxes can be one, or the same as the number of traversable devices, which is not limited in the embodiments of the present application. If there are multiple traversable devices, the number of bubble boxes is multiple, each of the multiple bubble boxes can correspond to a traversable device, and each bubble box can display the icon and / or model of the corresponding traversable device. The user can click on the bubble box to select a traversable device from multiple traversable devices for keyboard and mouse traversal, or can select a traversable device from multiple traversable devices for keyboard and mouse traversal by clicking on the icon or model of the traversable device. This is not limited in the embodiments of the present application. If there are multiple traversable devices, the number of bubble boxes is one, and this bubble box can display the icons and / or models of multiple traversable devices. The user can select a traversable device from multiple traversable devices for keyboard and mouse traversal by clicking on the icon or model of the traversable device.

[0173] Figure 8 Only one example of a bubble box is shown. The embodiments of the present application will not provide further examples for the case where the bubble box does not display the icon and / or model of the traversable device, or where multiple bubble boxes are displayed when there are multiple traversable devices, as well as other display forms of the bubble box.

[0174] In one possible implementation, each time the computer 11 detects that the mouse pointer is at the edge of the computer screen, a bubble box corresponding to the current traversable device is displayed.

[0175] In another possible implementation, when the computer 11 detects the mouse pointer at the edge side1 of the computer screen for the first time, the bubble box corresponding to the current traversable device is displayed. After the user selects a traversable device, when the computer 11 detects the mouse pointer at the edge side1 of the computer screen again, the bubble box corresponding to the current traversable device is no longer displayed; and when the computer 11 detects the mouse pointer at the edge side2 of the computer screen, the bubble box corresponding to the current traversable device is displayed. In this implementation, when the computer 11 detects the mouse pointer at the edge side2 of the computer screen, the bubble boxes corresponding to all the current traversable devices can be displayed, or only the bubble boxes corresponding to the traversable devices that are not currently selected can be displayed. For example, the current traversable devices are device 1, device 2, and device 3. If the mouse pointer of computer 11 is at the right edge of the computer screen and the user selects the bubble box corresponding to device 1, it means that the user has set device 1 as the crossing device on the right side of computer 11, that is, by sliding the mouse of computer 11 to the right, the mouse can reach the screen of device 1; when the mouse pointer of computer 11 is at the left edge of the computer screen, computer 11 displays the bubble boxes corresponding to device 2 and device 3 on the left edge of the computer screen for the user to select; if the user selects device 2, it means that the user has set device 2 as the crossing device on the left side of computer 11, that is, by sliding the mouse of computer 11 to the left, the mouse can reach the screen of device 2.

[0176] It should be noted that the crossing direction of the keyboard and mouse may be related to the position between the devices, or may be unrelated to the position between the devices. This is not limited in the embodiments of the present application. For example, in the case where the crossing direction of the keyboard and mouse is related to the position between the devices, if device 1 is on the left side of device 2, the mouse of device 2 slides to the left, and the mouse can reach the screen of device 1. In the case where the crossing direction of the keyboard and mouse is unrelated to the position between the devices, if device 1 is on the left side of device 2, but the crossing direction of device 1 can be set to the right side of device 2 by manually establishing communication or manually setting the crossing direction, that is, the mouse of device 2 slides to the right, the mouse can reach the screen of device 1.

[0177] When a keyboard and mouse connection has been established between terminal devices, keyboard and mouse traversal between terminal devices can be achieved. The following will introduce the PC-centric keyboard and mouse traversal method. Among them, in the PC-centric keyboard and mouse traversal method, the PC usually initiates the keyboard and mouse connection, and the PC is the communication center. The PC can connect to one or more Android devices that can be traversed. In other words, the keyboard and mouse traversal between devices must pass through the PC. The PC-centric keyboard and mouse traversal method can be applied to Figure 1 In the communication system shown in FIG. Figure 1 As shown, the computer 11 is connected to the mobile phone 12 and the tablet 13 respectively. At this time, the computer 11 serves as the communication center of the mobile phone 12 and the tablet 13.

[0178] In the embodiment of the present application, in the PC-centric crossing mode, when the computer 11 initiates a keyboard and mouse crossing event, and the keyboard and mouse of the computer 11 cross to the mobile phone 12 or the tablet 13, the computer 11 can be called the source device or the initiator, and the mobile phone 12 or the tablet 13 can be called the opposite device or the receiving end. When the mobile phone 12 initiates a keyboard and mouse crossing event, and the keyboard and mouse of the mobile phone 12 cross to the computer 11 or the tablet 13, the mobile phone 12 can be called the source device or the initiator, and the computer 11 or the tablet 13 can be called the opposite device or the receiving end. However, the embodiment of the present application is not limited to this.

[0179] The PC-centric keyboard and mouse crossover can be divided into two modes: In the first crossover mode, the operation task is mainly on the PC side, and the Android device needs to be operated. In this mode, the PC keyboard and mouse can be crossed to the Android device. For example, Figure 1 In the second crossing mode, the operation task is mainly on the Android device side, and the PC needs to be operated. In this mode, the keyboard and mouse of the Android device can be crossed to the PC. For example, Figure 1 The keyboard and mouse of the mobile phone 12 or the keyboard and mouse of the tablet 13 are transferred to the screen of the computer 11.

[0180] The two crossing methods are introduced below.

[0181] As an example of the first method (crossing from a PC keyboard and mouse to an Android device), see Fig. 9 , is a schematic diagram of a keyboard and mouse traversal scenario provided in an embodiment of the present application. Fig. 9 The keyboard and mouse crossing scene shown can be applied to Figure 1 In the communication system shown in FIG. Figure 1 As shown, the right side of the computer 11 is connected to the mobile phone 12, and the left side is connected to the tablet 13.

[0182] like Fig. 9 As shown in (a) of FIG. 1 , when the user manipulates the mouse of the computer 11 to move the mouse pointer 111 on the screen of the computer 11 to the right, as shown in FIG. Fig. 9 As shown in (a) and (b) of FIG. 1 , the computer 11 can realize keyboard and mouse crossover with the mobile phone 12. In other words, the mouse pointer 111 displayed on the computer 11 can cross from the screen of the computer 11 to the screen of the mobile phone 12 to the right. When the mouse pointer 111 of the computer 11 crosses to the screen of the mobile phone 12, as shown in FIG. Fig. 9 As shown in (b) in FIG. 1 , the mouse pointer 111 may be displayed in the form of a circle on the screen of the mobile phone 12. Fig. 9As shown in (b) in FIG. 1 , the user can also operate the mouse of the computer 11 to move the mouse pointer 111 on the screen of the mobile phone 12 to the left, as shown in FIG. Fig. 9 As shown in (b) and (c) in FIG. 1 , the mouse pointer 111 on the screen of the mobile phone 12 moves back to the screen of the computer 11 to the left, as shown in FIG. Fig. 9 As shown in (c) in FIG. 1 , the mouse pointer 111 is displayed in the form of an arrow on the screen of the computer 11 .

[0183] In some implementations, when the mouse of the computer 11 crosses to the screens of different Android devices, the mouse pointer 111 may be displayed in the same form on the screens of the different Android devices. Figure 1 In the communication system shown in , the computer 11 can also realize keyboard and mouse traversal with the tablet 13. In other words, the mouse pointer 111 displayed by the computer 11 can traverse from the screen of the computer 11 to the left to the screen of the tablet 13. When the mouse pointer 111 of the computer 11 traverses to the screen of the tablet 13, the mouse pointer 111 can also be displayed in the form of a circle on the screen of the tablet 13.

[0184] In other implementations, when the mouse of the computer 11 passes to the screen of different Android devices, the mouse pointer 111 may be displayed in different forms on the screen of different Android devices. For example, when the mouse pointer 111 of the computer 11 passes to the screen of the tablet 13, the mouse pointer 111 may be displayed in the form of a diamond on the screen of the tablet 13.

[0185] In this example, the external mouse of computer 11 is displayed as an arrow on the screen of computer 11, while the external mouse of mobile phone 12 is displayed as a circle on the screen of mobile phone 12. The mouse pointer 111 passes from the screen of computer 11 to the screen of mobile phone 12 and is displayed as a circle on the screen of mobile phone 12; the mouse pointer 111 passes from the screen of mobile phone 12 back to the screen of computer 11 and is displayed as an arrow on the screen of computer 11 again. It should be noted that in other examples, after the mouse pointer passes from the computer to the mobile phone, it can also be displayed in other forms, which is not limited here.

[0186] After the keyboard and mouse of the computer 11 are transferred to the mobile phone 12 or the tablet 13, the user can use the mouse to click, drag or double-click the mobile phone 12 or the tablet 13, and can use the keyboard to input information on the mobile phone 12 or the tablet 13.

[0187] See also Fig.10 , is a schematic diagram of an interaction scenario provided by an embodiment of the present application. Fig.10 In the example in FIG. 1 , after the mouse pointer 111 of the computer 11 crosses the screen of the mobile phone 12, the user can move the position of the mouse pointer 111 on the mobile phone screen by operating the mouse. Fig.10As shown in (a) of FIG. 1 , the mouse pointer 111 is located on the “memo” application icon. The user can click the “memo” application icon through the mouse pointer 111, and the mobile phone 12 can display the following information in response to the user clicking the “memo” application icon: Fig.10 The interface shown in (b) in the figure. Fig.10 As shown in (b) of FIG. 1 , the mobile phone 12 displays a memo interface. The user can input information "memo information" on the memo interface through the keyboard, and the mobile phone 12 displays the text "memo information" on the memo interface in response to the user inputting information on the memo interface.

[0188] In order to better understand the embodiments of the present application, Figure 2 and Figure 3 The software architecture shown introduces the implementation principle of the first crossing method (crossing the keyboard and mouse of the PC to the Android device).

[0189] See also Fig.11 , is a schematic diagram of the implementation principle of a traversal method provided in an embodiment of the present application. Fig.11 As shown, the implementation principle of the PC keyboard and mouse passing through to the Android device may include the following steps:

[0190] Step 000: The input driver of the Windows system detects the keyboard and mouse events of the PC and reports the keyboard and mouse events of the Windows system to the input framework of Windows.

[0191] Step 001: The input framework of the Windows system reports the received keyboard and mouse events to the keyboard and mouse service module of the Windows system.

[0192] In the embodiment of the present application, the input framework of the Windows system can obtain the keyboard and mouse event, which includes the keyboard and mouse displacement (or position offset). The input framework of the Windows system reports the keyboard and mouse event to the keyboard and mouse service module of Windows. The keyboard and mouse service module of Windows determines whether the keyboard and mouse crossing has occurred according to the keyboard and mouse displacement.

[0193] Step 002: If the keyboard and mouse service module of the Windows system recognizes that keyboard and mouse crossing has occurred (i.e., the keyboard and mouse of the PC crosses to the Android device) based on the received keyboard and mouse events, an indication message of intercepting the event is sent to the input framework of the Windows system.

[0194] Intercept events are used to intercept keyboard and mouse events. The indication information of intercept events is used to instruct the input framework to set keyboard hooks, mouse hooks, and shortcut hooks. Among them, hooks are a platform for Windows message processing mechanisms, on which applications can set subroutines to monitor certain messages of specified windows, and the monitored windows can be created by other processes. When a message arrives, it is processed before the target window processing function. The hook mechanism allows applications to intercept and process window messages or specific events. A hook is actually a program segment for processing messages, which is hung into the system through system calls. Whenever a specific message is sent, before it reaches the destination window, the hook program captures the message first, that is, the hook function gets control first. At this time, the hook function can process (change) the message, or continue to pass the message without processing it, or force the message to end.

[0195] The keyboard hook is used to obtain the user's input operation through the keyboard. For example, the keyboard hook is used to obtain the user's operation of inputting "memo information" through the keyboard. The mouse hook is used to obtain the user's control operation through the mouse. For example, the mouse hook is used to obtain the user's click or drag operation through the mouse. The shortcut key hook is used to obtain the user's shortcut key operation. For example, the shortcut key hook is used to obtain the user's Ctrl+C, Ctrl+V, ATL+TAB or Ctrl+Alt+

[0196] DELETE operation.

[0197] By intercepting events, the mouse passes from the PC to the Android device, and the input framework of the Windows system intercepts the monitored keyboard and mouse events, so that the business on the PC does not respond to keyboard and mouse events, so that the PC does not affect the mouse events.

[0198] It should be noted that the input framework of the Windows system has the ability to distribute events and input data. Among them, data input refers to the Windows input framework reporting the relevant data of the event to the business module of the application layer. Event distribution refers to the Windows input framework reporting the event to the business module of the application layer. For the application layer, if a business module connected to the application layer receives the event reported by the input framework and the relevant data of the event, it can respond to the event according to the relevant data of the event. For example, the relevant data of the keyboard and mouse event includes the keyboard and mouse displacement. After a business module of the application layer responds to the keyboard and mouse event, it displays the movement of the keyboard and mouse and the position reached on the display screen of the PC according to the keyboard and mouse displacement. If a business module of the application layer receives the relevant data of the event but does not receive the event, it cannot respond to the event. In other words, intercepting the event is equivalent to notifying each business module of the application layer of the Windows system not to respond to the keyboard and mouse event, but does not affect the reporting of the data (keyboard and mouse displacement) of the keyboard and mouse event.

[0199] Step 003: The keyboard and mouse service module of the Windows system packages and processes the keyboard and mouse events according to a preset protocol to obtain a data packet of the keyboard and mouse events.

[0200] The keyboard and mouse events are packaged according to the preset protocol, and the relevant data involved in the keyboard and mouse events can be converted into data in a corresponding format according to the data format defined in the preset protocol, and then the data after the conversion is packaged.

[0201] Optionally, the keyboard and mouse service module of the Windows system can encrypt data packets to prevent leakage of keyboard and mouse events.

[0202] Step 004: The keyboard and mouse service module of the Windows system sends the data packet of the keyboard and mouse event to the communication module of the Windows system.

[0203] Optionally, the keyboard and mouse service module of the Windows system can send the data packet of the keyboard and mouse event and the keyboard and mouse crossover indication information to the communication module of the Windows system. The keyboard and mouse crossover indication information is used to instruct the keyboard and mouse service module of the Android system to perform keyboard and mouse crossover.

[0204] Step 005: The communication module of the Windows system sends the data packet of the keyboard and mouse events to the communication module of the Android system.

[0205] Step 006: The communication module of the Android system reports the received data packet to the keyboard and mouse service module of the Android system.

[0206] Step 007: The keyboard and mouse service module of the Android system parses the data packet according to a preset protocol to obtain keyboard and mouse events.

[0207] As described in step 003, if the keyboard and mouse service module of the Windows system encrypts the data packet, the keyboard and mouse service module of the Android system first decrypts the encrypted data packet to obtain the data packet, and then parses the data packet according to the preset protocol to obtain the keyboard and mouse event. The data parsed by the keyboard and mouse service module of the Android system includes the keyboard and mouse event and its related data (such as keyboard and mouse displacement).

[0208] It should be noted that the preset protocol used by the keyboard and mouse service module of the Android system for parsing is consistent with the preset protocol used by the keyboard and mouse service module of the Windows system for packaging. In some application scenarios, the preset protocol can be shared after the PC and the Android device establish a keyboard and mouse connection.

[0209] Step 008: The keyboard and mouse service module of the Android system injects the keyboard and mouse events into the virtual input driver of the Android system.

[0210] Step 009: The virtual input driver of the Android system reports the keyboard and mouse events to the input framework of the Android system.

[0211] The input framework of the Android system can report keyboard and mouse events to various business modules in the Android application layer to implement keyboard and mouse functions.

[0212] Through the above steps, keyboard and mouse can be crossed between PC and Android device.

[0213] It should be noted that the prerequisite for achieving keyboard and mouse crossover between PC and Android device is that the keyboard and mouse connection has been established between PC and Android device. Figure 4 The contents described in the embodiments will not be repeated here.

[0214] In order to simplify the steps, the process of data interaction between the terminal devices through the communication module is omitted in the following embodiments, as well as the process of data interaction between the keyboard and mouse management submodule and the communication module through the connection scheduling submodule. But it can be understood that the data interaction between the keyboard and mouse service module of the PC and the keyboard and mouse service module of the Android device (such as the data packet of the interactive keyboard and mouse event) can be understood as the keyboard and mouse service module of the PC first transmitting the data to the communication module of the PC, and then transmitting it to the communication module of the Android device by the communication module of the PC, and transmitting it to the keyboard and mouse service module of the Android device by the communication module of the Android device. In addition, the keyboard and mouse service module is described as a whole in the following embodiments, but it can be understood that the interaction between the keyboard and mouse service module and the communication module all needs to be through the connection scheduling submodule in the keyboard and mouse service module, that is, the other submodules in the keyboard and mouse service module except the connection scheduling submodule first transmit the data to the connection scheduling submodule, and then transmit it to the communication module by the connection scheduling submodule, or the communication module first transmits the data to the connection scheduling submodule, and then transmits it to the other submodules in the keyboard and mouse service module by the connection scheduling submodule.

[0215] Based on the above implementation principles, the first crossing method (crossing the PC keyboard and mouse to the Android device) is introduced below.

[0216] See also Fig.12 , is a schematic diagram of an interactive process of a keyboard and mouse traversal method provided in an embodiment of the present application. Fig.12 As shown, the interaction process of the keyboard and mouse of the PC passing through to the Android device may include the following steps:

[0217] S1201, establish a keyboard and mouse connection between the PC and the Android device.

[0218] For the keyboard and mouse connection in this step, please refer to Figure 4 The description in the embodiments will not be repeated here.

[0219] S1202, the keyboard and mouse service module of the PC registers a monitoring event of the keyboard and mouse events with the input framework of the PC.

[0220] After the keyboard and mouse service module of the PC registers the monitoring events of the keyboard and mouse time, it can monitor the keyboard and mouse events.

[0221] S1203: After registering the monitoring event, the input framework of the PC continues to report the keyboard and mouse displacement to the keyboard and mouse service module of the PC.

[0222] For example, when the user presses the keyboard, moves, clicks, or drags the mouse, the PC input framework can obtain the keyboard and mouse events and the keyboard and mouse displacement (or position offset) corresponding to the current keyboard and mouse events through the PC input driver.

[0223] S1204: The keyboard and mouse service module of the PC determines whether the current mouse pointer reaches the edge of the PC screen according to the keyboard and mouse displacement.

[0224] It is understandable that S1203-S1204 can be performed continuously. In other words, after registering the monitoring event, the input framework of the PC continuously reports the keyboard and mouse displacement to the keyboard and mouse service module of the PC, and each time the keyboard and mouse service module of the PC receives the keyboard and mouse displacement, it determines whether the current keyboard and mouse have reached the edge of the PC screen based on the current keyboard and mouse displacement.

[0225] The process of determining the positions of the keyboard and mouse may refer to the description in the following S1301 - S1305 embodiments.

[0226] S1205: If the current keyboard and mouse reach the edge of the PC screen, the keyboard and mouse service module of the PC obtains the crossing position of the keyboard and mouse.

[0227] If the current keyboard and mouse have not reached the edge of the PC screen, the keyboard and mouse service module of the PC continues to monitor the keyboard and mouse events.

[0228] In this step, the keyboard and mouse service module of the PC can determine the crossing position according to the keyboard and mouse displacement reported by the input framework.

[0229] S1206, sending a crossing instruction to the keyboard and mouse service module of the Android device.

[0230] Correspondingly, the keyboard and mouse service module of the Android device receives the pass-through instruction. The keyboard and mouse service module of the PC informs the keyboard and mouse service module of the Android device of the keyboard and mouse pass-through through the pass-through instruction.

[0231] S1207, after receiving the pass-through instruction, the keyboard and mouse service module of the Android device replies with confirmation information to the keyboard and mouse service module of the PC.

[0232] S1208, the keyboard and mouse service module of the Android device determines the crossing position based on the crossing instruction.

[0233] In one implementation, the traversal instruction may include the traversal position of the keyboard and mouse on the PC screen. If the screen resolutions of the Android device and the PC are different, there will be differences in the position of the mouse on the screen when the mouse traverses between the two devices. The keyboard and mouse service module of the Android device may determine the traversal position of the mouse on the Android device screen based on the position of the mouse on the PC screen.

[0234] For example, if the mouse pointer's crossing position on the PC screen is (x1, y1), and the crossing direction is left crossing or right crossing, the PC's conversion module can be based on s y Change the transformation of y1, that is, y1×s y The value of the horizontal axis is a preset value, which is the minimum or maximum value of the screen side length on the X-axis. R 1y is the side length of the y-axis of the Android device screen, R 2y The length of the y-axis of the PC screen.

[0235] If the mouse traversal position on the PC screen is (x1, y1), and the traversal direction is up or down, the PC conversion module can be based on s x Change the transformation of x1, that is, x1×s x The value of the vertical coordinate is a preset value, which is the minimum or maximum value of the screen side length on the X-axis. R 1x is the length of the x-axis of the Android device screen, R 2x The length of the x-axis of the PC screen.

[0236] For example, in Fig.13 In the example shown, assuming that the screen resolution of the PC is 1920×1080, the crossing position is (1920,960), and the crossing direction is right crossing, the converted position can be (0, 960×s y ). If the crossing position is (0,960) and the crossing direction is right crossing, and the screen resolution of the Android device is 2000×1000, the converted position can be (2000, 960×s y ).in,

[0237] It is understandable that, in another implementation, the crossing instruction may include the proportion of the crossing position of the keyboard and mouse on the PC screen to the screen side length, that is, The keyboard and mouse service module of the Android device calculates the keyboard and mouse crossing positions on the screen of the Android device according to the received ratio and the side length of the screen of the Android device.

[0238] It should be noted that the above x and y It can be calculated by the keyboard and mouse service module of the PC or Android device after the Android device establishes a keyboard and mouse connection with the PC. The data can be exchanged through the crossing command.

[0239] S1209, the keyboard and mouse service module of the Android device updates the crossing status based on the crossing instruction.

[0240] Updating the crossing state in this step refers to updating the crossing state from not crossing to crossing.

[0241] S1210, after receiving the confirmation information, the keyboard and mouse service module of the PC sends an indication information of intercepting the event to the input framework of the PC.

[0242] The interception event in this step is Fig.11 The function of step 002 in the embodiment is the same as that of step 003. Fig.11 The description in the embodiments will not be repeated here.

[0243] It should be noted that S1208 - S1210 are not sequentially ordered and can be executed in parallel.

[0244] S1211, the keyboard and mouse service module of the PC packages the reported keyboard and mouse events according to a preset protocol to obtain a data packet of the keyboard and mouse events.

[0245] S1212, the keyboard and mouse service module of the PC sends the data packet to the keyboard and mouse service module of the Android device.

[0246] S1213, the keyboard and mouse service module of the Android device responds to the keyboard and mouse events.

[0247] Specifically, the keyboard and mouse service module of the Android device parses the data packet, obtains the keyboard and mouse displacement of the keyboard and mouse event, updates the keyboard and mouse position according to the keyboard and mouse displacement, and injects the keyboard and mouse event into the virtual input driver of the Android device according to the updated keyboard and mouse position; the virtual input driver of the Android device reports the keyboard and mouse event to the input framework of the Android device, and the input framework of the Android device distributes the keyboard and mouse event to various service modules of the application layer of the Android device to realize the functions of PC keyboard and mouse on the Android device.

[0248] Steps S1211-S1213 and Fig.11 The principles of steps 003-008 in the embodiment are the same, see Fig.11 The description in the embodiments, Fig.12 No other indication is given.

[0249] S1214, the keyboard and mouse service module of the Android device monitors the keyboard and mouse crossing.

[0250] In an embodiment of the present application, the process of the keyboard and mouse service module of the Android device monitoring the keyboard and mouse crossing includes: the keyboard and mouse service module of the Android device continuously receives the data packets sent by the keyboard and mouse service module of the PC, parses the keyboard and mouse events according to the data packets, and injects the keyboard and mouse events into the virtual input driver; the virtual input driver reports the keyboard and mouse events to the input framework of the Android device; the input framework reports the keyboard and mouse events to various service modules of the application layer of the Android device, including the keyboard and mouse service module; the keyboard and mouse service module obtains the keyboard and mouse displacement according to the reported keyboard and mouse events.

[0251] It is understandable that during the period when the keyboard and mouse of the PC are passed to the Android device, S1211-S1214 can be performed continuously. In other words, during the period when the keyboard and mouse of the PC are passed to the Android device, the keyboard and mouse service module of the PC continuously sends data packets to the keyboard and mouse service module of the Android device, and the keyboard and mouse service module of the Android device continuously monitors the keyboard and mouse passing, and whenever a data packet is received, it performs corresponding processing according to the keyboard and mouse events parsed from the data packet.

[0252] Since the keyboard and mouse service module of the PC continuously sends data packets to the keyboard and mouse service module of the Android device, it can be more vividly understood here that the keyboard and mouse service module of the PC sends a data stream to the keyboard and mouse service module of the Android device.

[0253] After the keyboard and mouse are passed to the Android device, they can be passed back. In the scenario where the keyboard and mouse are passed back to the PC, the operation of keyboard and mouse passing can be understood as the operation of passing back, that is, passing from the Android device back to the PC, which is explained below.

[0254] S1215, the keyboard and mouse service module of the Android device determines whether the current keyboard and mouse have reached the edge of the screen of the Android device according to the monitored keyboard and mouse displacement.

[0255] The process of determining the positions of the keyboard and mouse may refer to the description in the following S1301 - S1305 embodiments.

[0256] S1216: If the current keyboard and mouse reach the edge of the screen of the Android device, a message to end the crossing is sent to the keyboard and mouse service module of the PC.

[0257] S1217, the keyboard and mouse service module of the PC receives the information of ending the crossing, and sends a stop instruction of stopping the interception event to the input framework of the PC.

[0258] The input framework of the PC receives the stop instruction of the interception event and stops intercepting the keyboard and mouse events. After stopping intercepting the keyboard and mouse events, the input framework of the PC redistributes the keyboard and mouse events to each business module in the application layer of the PC.

[0259] S1218, the keyboard and mouse service module of the PC is restored.

[0260] In this step, restoring the state includes updating the crossing state and restoring the keyboard and mouse positions. Wherein, updating the crossing state means updating the crossing state from crossing to not crossing.

[0261] In one implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the PC can be restored according to the positions of the keyboard and mouse on the PC screen that are transferred back from the Android device.

[0262] In another implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the PC can be restored according to the keyboard and mouse positions that are crossed from the PC to the screen of the Android device.

[0263] In another implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the PC can be restored to a preset position, for example, the preset position is the center of the PC screen.

[0264] S1219, the keyboard and mouse service module of the PC sends a setting completion message to the keyboard and mouse service module of the Android device.

[0265] Correspondingly, the keyboard and mouse service module of the Android device receives the information of setting completion, which is used to indicate that the keyboard and mouse crossing has ended.

[0266] It should be noted that in the embodiment of the present application, there is no limitation on the order of S1219 and S1217, and the two can be executed in parallel.

[0267] S1220, the keyboard and mouse service module of the Android device updates the crossing status based on the setting completion information.

[0268] In this step, updating the crossing state means updating the crossing state from crossing to not crossing.

[0269] The keyboard and mouse crossing method provided in the embodiment of the present application can adjust the keyboard and mouse crossing positions based on different devices after the keyboard and mouse cross-device crossing is achieved. After the mouse cross-device crossing is successful, the keyboard and mouse events are intercepted at the PC and responded to on the Android device side, thereby achieving keyboard and mouse cross-device crossing.

[0270] In the above method, S1204 and S1215 involve the determination of the position of the keyboard and mouse. A method for detecting the position of the mouse is introduced below. The method for detecting the position of the mouse may include the following steps:

[0271] S1301. The keyboard and mouse service module obtains the current position of the mouse.

[0272] The current position of the mouse is the coordinate of the mouse on the screen. The keyboard and mouse service module can create a two-dimensional coordinate system on the screen, and the coordinate of the mouse in the two-dimensional coordinate system is the current position of the mouse.

[0273] For example, Fig.13 A schematic diagram of establishing a screen coordinate system is shown. Fig.13 As shown, the screen OABC establishes a coordinate system with O as the origin, OA as the positive direction of the X axis, and OC as the positive direction of the Y axis. The screen resolution is 1920×1080, the length of line segment OA is equal to the length of line segment CB (corresponding to 1920 pixels), and the length of line segment OC is equal to the length of line segment AB (corresponding to 1080 pixels). In the coordinate system, the coordinates of point O are (0,0), the coordinates of point A are (1920,0), the coordinates of point B are (1920,1080), and the coordinates of point C are (0,1080). The mouse position is the coordinate of the mouse in the coordinate system, for example, the mouse position can be (1920,960).

[0274] S1302: The keyboard and mouse service module obtains mouse displacement.

[0275] For example, when the user moves or drags the mouse, the virtual input driver of the PC reports keyboard and mouse events to the input framework of the PC. The input framework of the PC obtains the mouse displacement based on the keyboard and mouse events and reports the mouse displacement to the keyboard and mouse service module of the PC.

[0276] The mouse displacement is the distance the mouse moves. The mouse displacement can be represented by (Lx, Ly). Lx and Ly can be positive or negative values, which is not limited in the embodiment of the present application.

[0277] S1303. The keyboard and mouse service module calculates the position of the mouse after it moves based on the mouse position and mouse displacement.

[0278] For example, the current position of the mouse can be represented by (XRaw, YRaw), and the position of the mouse after it moves can be represented by (curX, curY). The position of the mouse after it moves can be represented by the following formula: curX=XRaw+Lx; curY=YRaw+Ly.

[0279] S1304: The keyboard and mouse service module determines whether the position of the mouse after movement exceeds the screen.

[0280] In the scenario where the mouse crosses the screen in the left and right directions, if curX is less than the minimum value of the screen on the X axis or greater than the maximum value of the screen on the X axis, the keyboard virtualization capability module of the PC can determine that the mouse is out of the screen.

[0281] In a scenario where the mouse moves across the screen in the up and down directions, if curY is less than the minimum value of the screen on the Y axis or greater than the maximum value of the screen on the Y axis, the keyboard virtualization capability module of the PC can determine that the mouse is out of the screen.

[0282] If the position of the mouse after moving exceeds the screen, it can be determined that the mouse has crossed, that is, S1305 is executed. If the position of the mouse after moving does not exceed the screen, the position of the mouse after moving can be determined as the current position, and the mouse movement is continued to be obtained, that is, S1302 is executed.

[0283] S1305: If the position of the mouse after movement exceeds the screen, the keyboard and mouse service module may determine that the mouse has crossed.

[0284] The keyboard and mouse service module can also determine the mouse's traversal direction. For example, in a scenario where the mouse traverses the screen left and right, the traversal direction can be left traversal or right traversal. In a scenario where the mouse traverses the screen up and down, the traversal direction can be up traversal or down traversal.

[0285] For example, in the above Fig.13 In the example shown, in the scenario where the mouse crosses the screen in the left and right directions, if curX is less than 0, the keyboard and mouse service module can determine that the mouse's crossing direction is left crossing. If curX is greater than 1920, the keyboard and mouse service module can determine that the mouse's crossing direction is right crossing. In the scenario where the mouse crosses the screen in the up and down directions, if curY is less than 0, the keyboard and mouse service module can determine that the mouse's crossing direction is up crossing. If curX is greater than 1080, the keyboard and mouse service module can determine that the mouse's crossing direction is down crossing.

[0286] It should be noted that the methods described in S1301-S1305 can be executed by a PC or an Android device. For example, in S1204, the methods in S1301-S1305 are executed by the keyboard and mouse service module of the PC. In S1215, the methods in S1301-S1305 are executed by the keyboard and mouse service module of the Android device.

[0287] The mouse crossing detection method provided in the embodiment of the present application can determine whether the mouse is beyond the screen and the crossing direction through the mouse position and mouse displacement, which is conducive to improving the accuracy of detecting mouse crossing.

[0288] As an example of the second method (Android device keyboard and mouse to PC), see Fig.14 , is a schematic diagram of another keyboard and mouse traversal scenario provided in an embodiment of the present application. Fig.14 The keyboard and mouse crossing scene shown can be applied to Figure 1 In the communication system shown in FIG. Figure 1 As shown, the right side of the computer 11 is connected to the mobile phone 12, and the left side is connected to the tablet 13.

[0289] like Fig.14 As shown in (a) of FIG. 1 , when the user operates the mouse of the mobile phone 12 to move the mouse pointer 121 on the screen of the mobile phone 12 to the left, as shown in FIG. Fig.14As shown in (a) and (b) of FIG. 1 , the keyboard and mouse can be crossed between the mobile phone 12 and the computer 11. In other words, the mouse pointer 121 displayed on the mobile phone 12 can cross from the screen of the mobile phone 12 to the left to the screen of the computer 11. When the mouse pointer 121 of the mobile phone 12 crosses to the screen of the computer 11, as shown in FIG. Fig.14 As shown in (b) of FIG. 1 , the mouse pointer 121 may be displayed in the form of an arrow on the screen of the computer 11. Fig.14 As shown in (b) of FIG. 1 , the user can also operate the mouse of the mobile phone 12 to move the mouse pointer 121 on the screen of the computer 11 to the right, as shown in FIG. Fig.14 As shown in (b) and (c) in FIG. 1 , the mouse pointer 121 on the screen of the computer 11 moves back to the screen of the mobile phone 12 to the right, as shown in FIG. Fig.14 As shown in (c) in FIG. 1 , the mouse pointer 121 is displayed in the form of a circle on the screen of the mobile phone 12 .

[0290] In this example, the mouse pointer 121 is displayed in the form of a circle on the mobile phone 12. The mouse pointer 121 passes from the mobile phone 12 to the computer 11 and is displayed in the form of an arrow on the computer 11. The mouse pointer 121 passes from the computer 11 back to the mobile phone 12 and is displayed in the form of a circle on the mobile phone 12 again. When the computer 11 is connected to an external mouse, the computer 11 also displays the mouse pointer in the form of an arrow. In other examples, after the mouse pointer passes from the mobile phone to the computer, it can also be displayed in other forms, which are not limited here.

[0291] After the keyboard and mouse of the mobile phone 12 are transferred to the computer 11, the user can use the mouse to click, drag or double-click the computer 11, and can use the keyboard to input information on the computer 11. No further illustration is given here.

[0292] In order to better understand the embodiments of the present application, Figure 2 and Figure 3 The software architecture shown introduces the implementation principle of the second crossing method (crossing the keyboard and mouse of the Android device to the PC).

[0293] See also Fig.15 , is a schematic diagram of the implementation principle of a traversal method provided in an embodiment of the present application. Fig.15 As shown, the implementation principle of transferring the keyboard and mouse of an Android device to a PC may include the following steps:

[0294] Step 010, the keyboard and mouse service module of the Windows system loads the virtual driver.

[0295] like Figure 2 and Fig.15As shown in , the driver layer of the Windows system includes a virtual driver and an input driver, wherein the virtual driver is used to drive the keyboard and mouse of the Android device, and the input driver is used to drive the keyboard and mouse of the PC. In this step, the keyboard and mouse service module of the Windows system loads the virtual driver to provide driver support for the subsequent keyboard and mouse of the Android device to pass through to the PC.

[0296] like Figure 2 As shown, the keyboard and mouse service module of the Windows system includes a virtual driver loading submodule. Accordingly, step 010 can load the virtual driver by the virtual driver loading submodule of the Windows system.

[0297] It should be noted that step 010 may be a step executed after the keyboard and mouse connection between the PC and the Android device is established, or may be a step executed before step 019. This is not limited.

[0298] Step 011, the virtual input driver of the Android system detects the keyboard and mouse events of the Android device, and reports the keyboard and mouse events of the Android system to the Android input framework.

[0299] Step 012: The input framework of the Android system reports the received keyboard and mouse events to the keyboard and mouse service module of the Android system.

[0300] In the embodiment of the present application, the input framework of the Android system can obtain the keyboard and mouse displacement (or position offset) corresponding to the keyboard and mouse event. The input framework of the Android system reports the keyboard and mouse event and the keyboard and mouse displacement corresponding to the keyboard and mouse event to the keyboard and mouse service module of Android. The keyboard and mouse service module of Android determines whether the keyboard and mouse crossing has occurred according to the keyboard and mouse displacement.

[0301] Step 013, if the keyboard and mouse service module of the Android system recognizes that keyboard and mouse crossing has occurred (i.e., the keyboard and mouse of the Android device crosses to the PC screen) based on the received keyboard and mouse events, the preset interface of the input framework of the Android system is called to intercept the keyboard and mouse event reporting.

[0302] The difference from Windows system is that Android system intercepts keyboard and mouse events by calling the preset interface of input framework, rather than intercepting events. By calling the preset interface, the mouse passes from Android device to PC, and the input framework of Android system intercepts the monitored keyboard and mouse events, so that the business on Android device does not respond to keyboard and mouse events, so that Android device does not affect keyboard and mouse events.

[0303] The principle / function of calling the preset interface of the input framework in the Android system is the same as the principle / function of the input framework executing interception events in the Windows system. For details, see Fig.11 The description of step 002 in the embodiment will not be repeated here.

[0304] Step 014, the keyboard and mouse service module of the Android system packages and processes the keyboard and mouse events according to a preset protocol to obtain a data packet of the keyboard and mouse events.

[0305] Optionally, the keyboard and mouse service module of the Android system can encrypt the data packet to prevent the leakage of keyboard and mouse events.

[0306] Step 015, the keyboard and mouse service module of the Android system sends the data packet of the keyboard and mouse event to the communication module of the Android system.

[0307] Optionally, the keyboard and mouse service module of the Android system can send the data packet of the keyboard and mouse event and the keyboard and mouse crossover indication information to the communication module of the Android system. The keyboard and mouse crossover indication information is used to instruct the keyboard and mouse service module of the Windows system to execute the keyboard and mouse crossover.

[0308] Step 016: The communication module of the Android system sends the data packet of the keyboard and mouse events to the communication module of the Windows system.

[0309] Step 017, the communication module of the Windows system reports the received data packet to the keyboard and mouse service module of the Windows system.

[0310] Step 018, the keyboard and mouse service module of the Windows system parses the data packet according to the preset protocol to obtain the keyboard and mouse events.

[0311] As described in step 014, if the keyboard and mouse service module of the Android system encrypts the data packet, the keyboard and mouse service module of the Windows system first decrypts the encrypted data packet to obtain the data packet, and then parses the data packet according to the preset protocol to obtain the keyboard and mouse event. The data parsed by the keyboard and mouse service module of the Windows system includes the keyboard and mouse event and its related data (such as keyboard and mouse displacement).

[0312] It should be noted that the preset protocol used by the keyboard and mouse service module of the Windows system for parsing is consistent with the preset protocol used by the keyboard and mouse service module of the Android system for packaging. In some application scenarios, the preset protocol can be shared after the PC and the Android device establish a keyboard and mouse connection.

[0313] Step 019, the keyboard and mouse service module of the Windows system injects the keyboard and mouse events into the virtual driver of the Windows system.

[0314] Step 020, the virtual driver of the Windows system reports the keyboard and mouse events to the input framework of the Windows system.

[0315] The input framework of the Windows system can report keyboard and mouse events to various business modules in the application layer of Windows to implement keyboard and mouse functions.

[0316] Through the above steps, keyboard and mouse can be crossed between Android devices and PC.

[0317] It should be noted that the premise of realizing keyboard and mouse crossover between Android devices and PC is that a keyboard and mouse connection has been established between PC and Android devices. Figure 4 The contents described in the embodiments will not be repeated here.

[0318] Based on the above implementation principle, the second method of crossing (keyboard and mouse crossing of Android device to PC) is introduced below. Fig.16 , is a schematic diagram of an interactive process of a keyboard and mouse traversal method provided in an embodiment of the present application. Fig.16 As shown, the interactive process of the keyboard and mouse of the Android device passing through to the PC may include the following steps:

[0319] S1601, establish a keyboard and mouse connection between the PC and the Android device.

[0320] For the keyboard and mouse connection in this step, please refer to Figure 4 The description in the embodiments will not be repeated here.

[0321] S1602, the keyboard and mouse service module of the Android device registers a listening event of the keyboard and mouse events with the input framework of the Android device.

[0322] After the keyboard and mouse service module of the Android device registers the monitoring events of the keyboard and mouse time, it can monitor the keyboard and mouse events.

[0323] S1603: After registering the monitoring event, the input framework of the Android device continuously reports the keyboard and mouse displacement to the keyboard and mouse service module of the Android device.

[0324] For example, when the user performs a pressing operation on the keyboard, or moves, clicks, or drags the mouse, the input framework of the Android device can obtain the keyboard and mouse events and the keyboard and mouse displacement (or position offset) corresponding to the current keyboard and mouse events through the virtual input driver of the Android device.

[0325] S1604, the keyboard and mouse service module of the Android device determines whether the current mouse pointer reaches the edge of the screen of the Android device according to the keyboard and mouse displacement.

[0326] It is understandable that S1603-S1604 can be performed continuously. In other words, after registering the monitoring event, the input framework of the Android device continuously reports the keyboard and mouse displacement to the keyboard and mouse service module of the Android device, and each time the keyboard and mouse service module of the Android device receives the keyboard and mouse displacement, it determines whether the current keyboard and mouse have reached the edge of the screen of the Android device based on the current keyboard and mouse displacement.

[0327] The process of determining the positions of the keyboard and mouse may refer to the description in the above-mentioned S1301-S1305 embodiments, which will not be described in detail here.

[0328] S1605: If the current keyboard and mouse reach the edge of the screen of the Android device, obtain the crossing position of the keyboard and mouse.

[0329] If the current keyboard and mouse have not reached the edge of the screen of the Android device, the keyboard and mouse service module of the Android device continues to monitor the keyboard and mouse events.

[0330] S1606: Send a pass-through instruction to the keyboard and mouse service module of the PC.

[0331] Correspondingly, the keyboard and mouse service module of the PC receives the pass-through instruction. The keyboard and mouse service module of the Android device informs the keyboard and mouse service module of the PC of the keyboard and mouse pass-through through the pass-through instruction.

[0332] S1607, after the keyboard and mouse service module of the PC receives the crossing instruction, if the current crossing status is not crossing, it replies confirmation information to the keyboard and mouse service module of the Android device.

[0333] S1608, the keyboard and mouse service module of the PC determines the crossing position based on the crossing instruction.

[0334] In this step, the method by which the keyboard and mouse service module of the PC determines the crossing position is the same as Fig.12 The principle of step S1208 in the embodiment is the same, and the details can be found in the description of S1208, which will not be repeated here.

[0335] S1609, the keyboard and mouse service module of the PC updates the crossing state based on the crossing instruction.

[0336] Updating the crossing state in this step refers to updating the crossing state from not crossing to crossing.

[0337] S1610, after receiving the confirmation information, the keyboard and mouse service module of the Android device calls the preset interface of the input framework of the Android device to intercept the keyboard and mouse event reporting.

[0338] The preset interface in this step is Fig.15 The function of step 013 in the embodiment is the same, see Fig.14 The description in the embodiments will not be repeated here.

[0339] It should be noted that S1608 - S1610 are not sequentially ordered and can be executed in parallel.

[0340] S1611, the keyboard and mouse service module of the Android device packages the reported keyboard and mouse displacements according to a preset protocol to obtain a data packet of the keyboard and mouse event.

[0341] S1612, the keyboard and mouse service module of the Android device sends the data packet to the keyboard and mouse service module of the PC.

[0342] S1613: The keyboard and mouse service module of the PC responds to the keyboard and mouse events.

[0343] Specifically, the keyboard and mouse service module of the PC parses the data packet, obtains the keyboard and mouse displacement of the keyboard and mouse event, updates the keyboard and mouse position according to the keyboard and mouse displacement, and injects the keyboard and mouse event into the virtual driver of the PC according to the updated keyboard and mouse position; the virtual driver of the PC reports the keyboard and mouse event to the input framework of the PC, and the input framework of the PC distributes the keyboard and mouse event to various service modules of the application layer of the PC to realize the keyboard and mouse functions of the Android device on the PC.

[0344] Steps S1611-S1613 and Fig.15 The principles of steps 014-020 in the embodiment are the same, see Fig.15 The description in the embodiments will not be repeated here.

[0345] S1614, the keyboard and mouse service module of the PC monitors the keyboard and mouse crossing.

[0346] In the embodiment of the present application, the process of the keyboard and mouse service module of the PC monitoring the keyboard and mouse crossing includes: the keyboard and mouse service module of the PC continuously receives the data packet sent by the keyboard and mouse service module of the Android device, parses the keyboard and mouse displacement corresponding to the keyboard and mouse event according to the data packet, updates the keyboard and mouse position according to the keyboard and mouse displacement, and injects the updated keyboard and mouse position into the virtual driver to update the keyboard and mouse position on the PC screen. When the keyboard and mouse service module of the PC monitors the keyboard and mouse crossing, the keyboard and mouse displacement of the keyboard and mouse event corresponding to the keyboard and mouse crossing can be obtained.

[0347] It is understandable that during the period when the keyboard and mouse of the Android device are passed to the PC, S1611-S1614 can be performed continuously. In other words, during the period when the keyboard and mouse of the Android device are passed to the PC, the keyboard and mouse service module of the Android device continuously sends data packets to the keyboard and mouse service module of the PC, and the keyboard and mouse service module of the PC continuously monitors the keyboard and mouse passing, and whenever a data packet is received, it performs corresponding processing according to the keyboard and mouse events parsed from the data packet.

[0348] Since the keyboard and mouse service module of the Android device continuously sends data packets to the keyboard and mouse service module of the PC, it can be more vividly understood here that the keyboard and mouse service module of the Android device sends a data stream to the keyboard and mouse service module of the PC.

[0349] After the keyboard and mouse are passed to the PC, they can be passed back. In the scenario where the keyboard and mouse are passed back to the Android device, the operation of keyboard and mouse passing can be understood as the operation of passing back, that is, passing from the PC back to the Android device, which is explained below.

[0350] S1615, the keyboard and mouse service module of the PC determines whether the current keyboard and mouse have reached the edge of the PC screen according to the monitored keyboard and mouse displacement.

[0351] The process of determining the positions of the keyboard and mouse may refer to the description in the above-mentioned S1301-S1305 embodiments, which will not be described in detail here.

[0352] S1616: If the current keyboard and mouse reach the edge of the PC screen, a message to end the crossing is sent to the keyboard and mouse service module of the Android device.

[0353] S1617, the keyboard and mouse service module of the Android device receives the information of ending the crossing, and stops calling the preset interface of the input framework of the Android device to stop intercepting the keyboard and mouse event reporting.

[0354] After the preset interface is stopped from being called, the input framework of the Android device stops intercepting the keyboard and mouse events. After the keyboard and mouse events are stopped from being intercepted, the input framework of the Android device redistributes the keyboard and mouse events to various business modules in the application layer of the Android device.

[0355] S1618, the keyboard and mouse service module of the Android device is restored to status.

[0356] In this step, restoring the state includes updating the crossing state and restoring the keyboard and mouse positions. Wherein, updating the crossing state means updating the crossing state from crossing to not crossing.

[0357] In one implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the Android device can be restored according to the positions of the keyboard and mouse on the screen of the Android device when the keyboard and mouse are returned from the PC.

[0358] In another implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the Android device can be restored according to the keyboard and mouse crossing positions from the Android device to the PC screen.

[0359] In another implementation of restoring the keyboard and mouse positions, the keyboard and mouse positions of the Android device can be restored according to a preset position. For example, the mouse position is restored to the center position of the Android device screen.

[0360] S1619, the keyboard and mouse service module of the Android device sends a setting completion message to the keyboard and mouse service module of the PC.

[0361] Correspondingly, the keyboard and mouse service module of the PC receives the information of setting completion, which is used to indicate that the keyboard and mouse crossing has ended.

[0362] It should be noted that in the embodiment of the present application, there is no limitation on the order of S1619 and S1617, and the two can be executed in parallel.

[0363] S1620, the keyboard and mouse service module of the PC updates the crossing status based on the setting completion information.

[0364] In this step, updating the crossing state means updating the crossing state from crossing to not crossing.

[0365] The keyboard and mouse crossing method provided in the embodiment of the present application can adjust the keyboard and mouse crossing positions based on different devices after the keyboard and mouse cross-device crossing is achieved. After the mouse cross-device crossing is successful, the keyboard and mouse events are intercepted at the Android device and responded to on the PC side, thereby achieving keyboard and mouse cross-device crossing.

[0366] like Fig.12 Examples and Fig.16 As shown in the embodiment, the similarities between the two are that the premise of both is that a keyboard and mouse connection has been established between the PC and the Android device, and the keyboard and mouse connection is centered on the PC. The difference between the two is that Fig.12 The PC is the initiator of the keyboard and mouse crossing event, and the Android device is the receiver; Fig.16 The Android device is the initiator of the keyboard and mouse crossing event, and the PC is the receiver.

[0367] Based on the above Figure 12 to Figure 16 The two keyboard and mouse traversal methods described in the embodiments enable users to flexibly operate multiple terminal devices through the same set of keyboard and mouse, avoiding switching operation modes back and forth between multiple terminal devices, simplifying operations, and improving user experience.

[0368] Above Fig. 9 and Fig.14 The application scenario shown is a keyboard and mouse traversal scenario between two terminal devices. In other application scenarios, the user may need to operate three or more terminal devices, which involves keyboard and mouse traversal between multiple terminal devices.

[0369] As an example of keyboard and mouse traversal between multiple terminal devices, the keyboard and mouse traversal between three terminal devices is used as an example. Fig.17 and Fig.18 , is a schematic diagram of a keyboard and mouse traversal scenario provided in an embodiment of the present application.

[0370] Fig.17 and Fig.18 The keyboard and mouse crossing scene shown can be applied to Figure 1 In the communication system shown in FIG. Figure 1 As shown, the right side of the computer 11 is connected to the mobile phone 12, and the left side is connected to the tablet 13.

[0371] like Fig.17 As shown in (a) in FIG. 1 , when the user manipulates the mouse pointer on the screen of the mobile phone 12 to move to the left, as shown in FIG. Fig.17 As shown in (a) and (b) in FIG. 1 , the keyboard and mouse can be crossed between the mobile phone 12 and the computer 11. In other words, the mouse pointer 121 displayed on the mobile phone 12 can cross from the screen of the mobile phone 12 to the left to the screen of the computer 11. Fig.17 As shown in (b) in FIG. 1 , when the user continues to manipulate the mouse of the mobile phone 12 to move the mouse pointer on the screen of the computer 11 to the left, as shown in FIG. Fig.17 As shown in (b) and (c) in the figure, keyboard and mouse can be crossed between the computer 11 and the tablet 13. In other words, the mouse pointer 121 displayed on the computer 11 can continue to cross from the screen of the computer 11 to the left to the screen of the tablet 13.

[0372] continue Fig.17 In the scene shown in (c) of FIG. 1 , the current mouse pointer 121 is on the screen of the tablet 13. Fig.18 As shown in (a) in FIG. 1 , when the user manipulates the mouse pointer on the screen of the tablet 13 to move to the right, as shown in FIG. Fig.18 As shown in (a) and (b) of FIG. 1 , the keyboard and mouse can be crossed between the tablet 13 and the computer 11. In other words, the mouse pointer 121 displayed on the tablet 13 can be crossed from the screen of the tablet 13 to the right to the screen of the computer 11. Fig.18 As shown in (b) in FIG. 1 , when the user continues to control the mouse pointer on the screen of the computer 11 to move to the right, the keyboard and mouse can be crossed between the computer 11 and the mobile phone 12. In other words, the mouse pointer 121 displayed on the computer 11 can be passed from the screen of the computer 11 back to the screen of the mobile phone 12, as shown in FIG. Fig.18 As shown in (c) in .

[0373] exist Fig.17 and Fig.18 In the application scenario shown, the mouse pointer of the mobile phone 12 is sent multiple times between the computer 11, the mobile phone 12 and the tablet 13, realizing the keyboard and mouse crossing across PCs.

[0374] In an application scenario where multiple Android devices implement keyboard and mouse crossover across PCs, the keyboard and mouse events detected by Android device I may be keyboard and mouse events of Android device I itself, in which case Android device I is in a non-crossover state; or they may be keyboard and mouse events of Android device II or PC crossover, in which case Android device I is in a crossover state. If no distinction is made, confusion of the crossover state may result.

[0375] Based on this consideration, an embodiment of the present application provides a keyboard and mouse traversal method, which can distinguish the keyboard and mouse traversal scenarios of multiple terminal devices, avoid the occurrence of confusion in the traversal state, and thus ensure the reliability of keyboard and mouse traversal.

[0376] like Fig.17 and Fig.18 In the application scenario shown, multiple keyboard and mouse traversal messages are sent between the computer 11, the mobile phone 12 and the tablet 13, which can be divided into two situations: in the first situation, the mouse pointer traverses from the screen of the PC to the screen of the Android device; in the second situation, the mouse pointer traverses from the screen of the Android device to the screen of the PC. The keyboard and mouse traversal methods in the two situations are introduced respectively below.

[0377] It should be noted that if Fig.17 and Fig.18 In the application scenario shown, keyboard and mouse traversal may occur between the PC and the two Android devices. Therefore, the premise of the following embodiments is that the PC has established keyboard and mouse connections with the two Android devices respectively. The method of establishing keyboard and mouse connections can be found in the above Figure 4 The description in the embodiments will not be repeated here.

[0378] The following describes the keyboard and mouse traversal method in the first case. Fig.19 , is a schematic diagram of the interaction flow of a keyboard and mouse traversal method provided in an embodiment of the present application. Fig.19 The method described can be applied in the case where the mouse pointer passes from the PC screen to the Android device screen. As an example and not limitation, Fig.19 As shown, the keyboard and mouse traversal method may include the following steps:

[0379] S1901, the keyboard and mouse service module of Android device I monitors the keyboard and mouse displacement of keyboard and mouse events.

[0380] The principle of this step is the same as that of S1214. For details, please refer to the description of S1214, which will not be repeated here.

[0381] S1902, the keyboard and mouse service module of Android device I determines whether the current mouse pointer reaches the edge of the screen based on the monitored keyboard and mouse displacement.

[0382] The method for determining whether the mouse pointer has reached the edge of the screen can be found in the description of the above S1301-S1305 embodiments, which will not be repeated here.

[0383] S1903: If the current mouse pointer reaches the edge of the screen, determine whether there is a traversable device at the edge of the current screen.

[0384] Exemplarily, on Android device 1, Fig.17In the case of the tablet 13 shown in the figure, since the Android device 1 is on the left side of the PC, there is no device on the left side of the Android device 1 that can be crossed. Therefore, if the current mouse pointer reaches the left edge of the screen, there is no need to consider the keyboard and mouse crossing; if the current mouse pointer reaches the right edge of the screen, S1904 is executed. Fig.17 In the case of the mobile phone 12 shown, since the Android device 1 is on the right side of the PC, there is no traversable device on the right side of the Android device 1. Therefore, if the current mouse pointer reaches the right edge of the screen, there is no need to consider keyboard and mouse traversal; if the current mouse pointer reaches the left edge of the screen, S1904 is executed.

[0385] S1904, if there is a traversable device at the edge of the current screen, determine whether the current mouse pointer belongs to the local keyboard and mouse (the keyboard and mouse of Android device I).

[0386] Depend on Fig.17 and Fig.18 As can be seen from the application scenario shown, the mouse pointer on the Android device screen may be the mouse pointer of its own keyboard and mouse, or it may be the mouse pointer passed through from the PC.

[0387] If the current mouse pointer belongs to the local keyboard and mouse (keyboard and mouse of Android device I), then Fig.14 The situation shown in the figure is that the keyboard and mouse of the Android device are passed to the PC. For the keyboard and mouse passing method, please refer to Fig.16 Steps S1605-S1614 in the embodiment are not described in detail here.

[0388] It should be noted that Fig.14 The application scenario shown only involves the keyboard and mouse traversal between a PC and an Android device. Fig.14 The application scenarios shown are different in that Fig.17 and Fig.18 In the application scenario shown, a situation in which keyboard and mouse crossover occurs between a PC and multiple Android devices. Therefore, before the keyboard and mouse of Android device I cross over to the PC, the keyboard and mouse of the PC may have already crossed over to Android device II. In this case, the PC is in a crossover state, that is, the input framework of the PC is executing an interception event to intercept the reporting of keyboard and mouse events.

[0389] In order to ensure that the PC can execute keyboard and mouse events after the keyboard and mouse of Android device I are passed through to the PC, in one embodiment, after S1606, the keyboard and mouse service module of the PC receives the passing instruction sent by Android device I, and then sends a stop instruction to the input framework of the PC to stop intercepting keyboard and mouse events, so as to instruct the input framework of the PC to stop intercepting keyboard and mouse events.

[0390] If the current mouse pointer does not belong to the local keyboard and mouse, it means that the current mouse pointer has passed through the screen of the PC, then step S1905 is executed.

[0391] In some embodiments, during the keyboard and mouse crossing process, the device ID of the device to which the keyboard and mouse belong can be packaged together with the keyboard and mouse events into a data packet. For example, when the keyboard and mouse of Android device 1 is crossed to PC, the keyboard and mouse service module of Android device 1 packages the device ID of Android device 1 together with the keyboard and mouse events into a data packet and sends it to the keyboard and mouse service module of PC. The keyboard and mouse service module of PC parses the keyboard and mouse events and device ID from the received data packet, and can confirm that the current keyboard and mouse is crossed from Android device 1 according to the device ID.

[0392] Accordingly, one implementation of determining whether it is a local keyboard and mouse may include:

[0393] The keyboard and mouse service module obtains the device ID corresponding to the current keyboard and mouse event; and determines whether it is a local keyboard and mouse based on the device ID.

[0394] It should be noted that when the Android device determines whether it is a local keyboard and mouse, the keyboard and mouse service module of the Android device performs the above determination step. When the PC device determines whether it is a local keyboard and mouse, the keyboard and mouse service module of the PC performs the above determination step.

[0395] In other implementations, during the keyboard and mouse crossover process, the device ID may be carried in each transmitted keyboard and mouse event data packet; the device ID may be carried in the first transmitted keyboard and mouse event data packet, and the device ID may no longer be carried in the subsequent transmitted keyboard and mouse event data packets. In other implementations, the device ID may also be carried in the crossover instruction sent when the keyboard and mouse crossover occurs for the first time.

[0396] In some other embodiments, the crossing direction of the keyboard and mouse can also be packaged into a data packet together with the keyboard and mouse events. For example, the keyboard and mouse of Android device I cross to PC, and Android device I is on the left side of PC, that is, the crossing direction is to the right. The keyboard and mouse service module of Android device I packages the crossing direction (to the right) and the keyboard and mouse events into a data packet and sends it to the keyboard and mouse service module of PC. The keyboard and mouse service module of PC parses the keyboard and mouse events and the crossing direction (to the right) from the received data packet, and can confirm that the current keyboard and mouse are crossed from Android device I according to the crossing direction.

[0397] like Fig.17 and Fig.18In the application scenario shown, the keyboard and mouse of Android device II first pass through to PC, and then pass through to Android device I. When the mouse pointer passes back from Android device I to PC, if it is judged only based on the crossing direction, it may be misjudged that the current keyboard and mouse belong to Android device I. In order to prevent misjudgment, in other embodiments, the crossing direction of the keyboard and mouse and the device ID to which the keyboard and mouse belong can also be packaged into a data packet together with the keyboard and mouse event. In this way, the terminal to which the keyboard and mouse belongs can be judged by the crossing direction, and the terminal to which the keyboard and mouse belongs can be judged from the device ID. The double judgment can effectively prevent the occurrence of misjudgment and help to more clearly distinguish different crossing situations.

[0398] In other implementations, it is also possible to determine whether it is a native keyboard and mouse or a virtual keyboard and mouse by calling a preset function. For example, the preset function can return a function value, and when the function value is false, it indicates a virtual keyboard and mouse; when the function value is true, it indicates a native keyboard and mouse.

[0399] S1905, if the current mouse pointer does not belong to the local keyboard and mouse (the keyboard and mouse of Android device I), the keyboard and mouse service module of Android device I sends a return instruction to the keyboard and mouse service module of the PC and ends the crossing.

[0400] The keyboard and mouse service module of Android device 1 can end the crossing according to steps S1617-S1619.

[0401] Correspondingly, the keyboard and mouse service module of the PC receives the return instruction. The keyboard and mouse service module of the Android device 1 informs the keyboard and mouse service module of the PC through the return instruction that the mouse pointer passes back to the PC.

[0402] S1906, the keyboard and mouse service module of the PC receives the return instruction and determines whether the currently traversing mouse pointer belongs to the local keyboard and mouse (the keyboard and mouse of the PC).

[0403] The principle of determining whether it is a local keyboard and mouse in this step is the same as that in S1904. For details, please refer to the description in S1904, which will not be repeated here.

[0404] If the mouse pointer currently traversing belongs to the local keyboard and mouse (PC keyboard and mouse), then Fig. 9 The situation shown in the figure is that the keyboard and mouse of the PC are passed from the Android device to the PC. For the keyboard and mouse passing method, please refer to Fig.16 Steps S1616-S1620 in the embodiment are not described in detail here.

[0405] If the currently traversed mouse pointer does not belong to the local keyboard and mouse (the keyboard and mouse of the PC), it means that the current mouse pointer is traversed from the Android device II, and then S1907 is executed.

[0406] It should be noted that, in the embodiment of the present application, when the keyboard and mouse service module of the PC determines that the current mouse pointer does not belong to the local keyboard and mouse, the keyboard and mouse service module of the PC can determine whether the currently traversed keyboard and mouse belongs to Android device I or Android device II according to the type of instruction received from Android device I. For example, if a traversal instruction is received, it means that the currently traversed keyboard and mouse belongs to Android device I; if a return instruction is received, it means that the currently traversed keyboard and mouse belongs to Android device II. Of course, it can also be determined collaboratively based on the device ID in the data packet.

[0407] S1907, the keyboard and mouse service module of the PC sends a switching notification to the keyboard and mouse service module of the Android device II.

[0408] S1908, the keyboard and mouse service module of the Android device II receives the switching notification and sends the data packet of the keyboard and mouse event to the keyboard and mouse service module of the PC.

[0409] Afterwards, the keyboard and mouse service module of the PC monitors the data packet of the keyboard and mouse event and executes steps S1613-S1615.

[0410] The situation corresponding to this step is Fig.14 The situation where the keyboard and mouse of the Android device are passed to the PC is the same. This step has the same principle as S1613-S1615. For details, please refer to the description of S1613-S1615, which will not be repeated here.

[0411] It should be noted that in order to simplify the steps, Fig.19 Only part of the software architecture in the PC and Android device is shown, which does not mean that other modules not shown do not participate in keyboard and mouse crossing. The modules participating in keyboard and mouse crossing can be found in the description of other embodiments.

[0412] In the embodiment of the present application, in the application scenario where the keyboard and mouse crossover occurs between multiple terminal devices, for the case where the mouse pointer crosses from Android device 1 to the PC screen, it is determined on the Android device 1 side whether it is a local keyboard and mouse, so that it can be distinguished whether the keyboard and mouse of Android device 1 crosses to the PC, or the keyboard and mouse of Android device 1 crosses back to the PC; in the case where the keyboard and mouse of Android device 1 crosses back to the PC, it is determined on the PC side whether it is a local keyboard and mouse, so that it can be distinguished whether the keyboard and mouse of Android device 1 crosses back to the PC, or the keyboard and mouse of Android device II crosses to the PC. Through the above method, the scenes of keyboard and mouse crossover of multiple terminal devices can be distinguished, and the situation of confusion in the crossover state can be avoided, thereby ensuring the reliability of keyboard and mouse crossover.

[0413] The following describes the keyboard and mouse traversal method in the second case. Fig. 20 , is a schematic diagram of the interaction flow of a keyboard and mouse traversal method provided in an embodiment of the present application. Fig. 20The method described can be applied to the situation where the mouse pointer passes from the screen of the Android device to the screen of the PC. As an example and not limitation, Fig. 20 As shown, the keyboard and mouse traversal method may include the following steps:

[0414] S2001, the keyboard and mouse service module of the PC monitors the keyboard and mouse displacement of the keyboard and mouse events.

[0415] The principle of this step is the same as that of S1614. For details, please refer to the description of S1614, which will not be repeated here.

[0416] S2002, the keyboard and mouse service module of the PC determines whether the current mouse pointer reaches the edge of the screen according to the monitored keyboard and mouse displacement.

[0417] The method for determining whether the mouse pointer has reached the edge of the screen can be found in the description of the above S1301-S1305 embodiments, which will not be repeated here.

[0418] S2003: If the current mouse pointer reaches the edge of the screen, determine whether there is a traversable device at the edge of the current screen.

[0419] For example, when there is a tablet 13 on the left side of the PC and no connected device on the right side, when the current mouse pointer on the PC screen reaches the left edge of the screen, there is a traversable device, tablet 13, then S2004 is executed; when the current mouse pointer on the PC screen reaches the right edge of the screen, there is no traversable device, and there is no need to consider keyboard and mouse traversal at this time.

[0420] S2004: If there is a traversable device at the edge of the current screen, determine whether the current mouse pointer belongs to the local keyboard and mouse (the keyboard and mouse of the PC).

[0421] Depend on Fig.17 and Fig.18 From the application scenario shown, it can be seen that the mouse pointer on the PC screen may be the mouse pointer of the native keyboard and mouse, or it may be the mouse pointer passed from the Android device.

[0422] If the current mouse pointer belongs to the local keyboard and mouse (PC keyboard and mouse), then Fig. 9 The scene shown in the figure shows the keyboard and mouse of the PC being transferred to the Android device. For the keyboard and mouse transfer method, please refer to Fig.12 Steps S1205-S1215 in the embodiment are not described in detail here.

[0423] It should be noted that Fig. 20, the mouse pointer passes from the left side of the PC screen to the Android device I; accordingly, steps S1205-S1215 are performed between the PC and the Android device I. It can be understood that if the mouse pointer passes from the right side of the PC screen to the Android device II, steps S1205-S1215 are performed between the PC and the Android device II.

[0424] If the current mouse pointer does not belong to the local keyboard and mouse (the keyboard and mouse of the PC), execute step S2005.

[0425] S2005: If the current mouse pointer does not belong to the local keyboard and mouse (the keyboard and mouse of the PC), determine whether the current mouse pointer belongs to the receiving end.

[0426] The receiving end represents the Android device corresponding to the current crossing direction. Figure 1 In the communication system shown, if the current mouse pointer crosses the screen of the computer 11 to the left, the left side of the computer 11 is the tablet 13, and the tablet 13 is the receiving end. If the current mouse pointer crosses the screen of the computer 11 to the right, the right side of the computer 11 is the mobile phone 12, and the mobile phone 12 is the receiving end.

[0427] If the current mouse pointer belongs to the receiving end, that is, the receiving end is the source end, Fig.14 The situation shown in the figure is that the keyboard and mouse of the Android device are passed from the PC to the Android device. For the keyboard and mouse passing method, please refer to Fig.16 Steps S1616-S1620 in the embodiment are not described in detail here.

[0428] The source end refers to the Android device to which the current mouse pointer belongs. For example, Figure 1 In the communication system shown, if the current mouse pointer belongs to the mobile phone 12, the mobile phone 12 is the source end, indicating that the keyboard and mouse of the mobile phone 12 first pass through to the computer 11, and then pass from the computer 11 to the tablet 13. If the current mouse pointer belongs to the tablet 13, the tablet 13 is the source end, indicating that the keyboard and mouse of the tablet 13 first pass through to the computer 11, and then pass from the computer 11 back to the tablet 13.

[0429] If the current mouse pointer does not belong to the receiving end, it means that the current mouse pointer is passed from the Android device II, and then S2006 is executed.

[0430] It should be noted that Fig. 20 The figure shows a situation where the mouse pointer passes from the left side of the PC screen to the Android device I. In this situation, the Android device I is the receiving end and the Android device II is the source end.

[0431] S2006, if the current mouse pointer does not belong to the receiving end (Android device I), a transfer instruction is sent to the source end (Android device II).

[0432] Accordingly, the source end can receive the transfer instruction, which is used to inform the source end that the receiving end of the keyboard and mouse event has changed.

[0433] This step represents the scenario where the keyboard and mouse are transferred from one Android device to another Android device via a PC. Figure 1 In the communication system shown, if the current mouse pointer on the screen of computer 11 belongs to the keyboard and mouse of mobile phone 12 (source end), the current mouse pointer passes to the left to tablet 13 (receiving end), which is equivalent to the receiving end of the keyboard and mouse passing through changing from computer 11 to tablet 13.

[0434] S2007, after receiving the transfer instruction, the keyboard and mouse service module of the source end (Android device II) sends a crossing instruction to the keyboard and mouse service module of the receiving end (Android device I).

[0435] In some implementations, the source end may send the traversal instruction to the receiving end through the PC, that is, the source end first sends the traversal instruction to the PC, which then forwards it to the receiving end.

[0436] In some other implementations, the source end may request to directly establish a keyboard and mouse connection with the receiving end. After the keyboard and mouse connection is established, the source end directly sends a crossing instruction to the receiving end.

[0437] S2008, after the receiving end (Android device I) receives the crossing instruction, if the current crossing status is not crossed, it replies confirmation information to the keyboard and mouse service module of the source end (Android device II).

[0438] In some scenarios, if the current pass-through state of the receiving end is pass-through, the keyboard and mouse service module of the source end will reply with occupation information to inform the source end that the current receiving end is in pass-through state. In this case, the source end can notify the user through the screen that the keyboard and mouse cannot pass through at this time.

[0439] It should be noted that in the embodiment of the present application, if a device is currently in a crossing state, keyboard and mouse crossing will no longer be performed between other devices.

[0440] S2009, the keyboard and mouse service module of the receiving end (Android device I) updates the crossing status based on the crossing instruction.

[0441] In this step, updating the crossing status means updating the crossing status from not crossing to crossing.

[0442] S2010, the keyboard and mouse service module of the receiving end (Android device I) determines the crossing position based on the crossing instruction.

[0443] In this step, the method for the receiving end to determine the crossing position can be referred to the description of the embodiments S1301-S1305.

[0444] S2011, after receiving the confirmation information, the keyboard and mouse service module of the source end (Android device II) packages and processes the keyboard and mouse events reported by the input framework of the source end (Android device II) according to the preset protocol to obtain a data packet of the keyboard and mouse events.

[0445] S2012, the source end (Android device II) sends the data packet of the keyboard and mouse events to the keyboard and mouse service module of the receiving end (Android device I).

[0446] Since in the embodiment of the present application, the communication system uses the PC as the communication center, in other words, the interaction between the Android device I and the Android device II must go through the PC. One implementation of this step is that the keyboard and mouse service module of the source end sends the data packet of the keyboard and mouse event to the communication module of the source end, and the communication module of the source end sends the data packet to the communication module of the PC; the communication module of the PC forwards the data packet to the communication module of the receiving end; the communication module of the receiving end sends the data packet to the keyboard and mouse service module of the receiving end.

[0447] In order to improve communication efficiency, another implementation method of step S2012 is that the source end and the receiving end establish a communication connection through a PC; after establishing the communication connection between the source end and the receiving end, the keyboard and mouse service module of the source end sends the data packet of the keyboard and mouse event to the communication module of the source end, and the communication module of the source end sends the data packet to the communication module of the receiving end; the communication module of the receiving end sends the data packet to the keyboard and mouse service module of the receiving end.

[0448] It should be noted that in order to simplify the schematic steps, Fig. 20 In S2012, the above data interaction process is omitted.

[0449] S2013, the keyboard and mouse service module of the receiving end (Android device I) responds to the keyboard and mouse events.

[0450] S2014, the keyboard and mouse service module of the receiving end (Android device I) monitors the keyboard and mouse crossing.

[0451] The principles of steps S2013-S2014 are the same as those of S1213-S1214. For details, please refer to the description of S1213-S1214, which will not be repeated here.

[0452] In some scenarios, there may be a communication failure between the source end and the receiving end, resulting in a disconnection of the communication between the source end and the receiving end. In this scenario, S2015 is executed.

[0453] S2015, if the communication between the source end and the receiving end is disconnected, the source end and the receiving end restore the status respectively.

[0454] Among them, the source end recovery state includes the source end stopping sending data packets of keyboard and mouse events to the receiving end, restoring the mouse pointer of the source end keyboard and mouse to the first preset position on the source end screen, and maintaining the communication connection with the PC. For example, the first preset position can be the center position of the source end screen, or the position of the initial crossing on the source end screen. The receiving end recovery state includes restoring the mouse pointer of the receiving end keyboard and mouse to the second preset position on the receiving end screen, and maintaining the communication connection with the PC. For example, the second preset position can be the center position of the receiving end screen, or the position of the mouse pointer on the receiving end screen before the keyboard and mouse crossing occurs. In other words, after the recovery state, the mouse pointers of the keyboard and mouse of the source end and the receiving end return to their respective screens.

[0455] It should be noted that in order to simplify the steps, Fig. 20 Only part of the software architecture in the PC and Android device is shown, which does not mean that other modules not shown do not participate in keyboard and mouse crossing. The modules participating in keyboard and mouse crossing can be found in the description of other embodiments.

[0456] It should be noted that Fig. 20 The case where Android device II is the source end and Android device I or PC is the receiving end is shown in the figure. The principle is similar to the case where Android device I is the source end and Android device II or PC is the receiving end, so it will not be repeated here.

[0457] In the embodiment of the present application, in the application scenario where keyboard and mouse traversal occurs between multiple terminal devices, for the case where the mouse pointer traverses from the PC screen to the screen of the Android device 1, it is determined on the PC side whether it is the local keyboard and mouse, so that it can be distinguished whether the PC keyboard and mouse traverse to the Android device 1, or the keyboard and mouse traverse from the PC back to the Android device 1, or the keyboard and mouse of the Android device II traverse to the Android device 1. Through the above method, the scenarios of keyboard and mouse traversal of multiple terminal devices can be distinguished, and the occurrence of confusion in the traversal state can be avoided, thereby ensuring the reliability of keyboard and mouse traversal.

[0458] In the embodiment of the present application, after the mouse pointer crosses, the mouse pointer after the crossing and the local mouse pointer can be combined into one mouse pointer, thereby avoiding the situation where the mouse pointer is confused.

[0459] In some application scenarios, conflicts may occur between the keyboard and mouse of the traversal and the local keyboard and mouse. Fig.21 , is a schematic diagram of a keyboard and mouse crossing conflict scenario provided in an embodiment of the present application. Fig.21 The scenario shown can be applied to Figure 1 In the communication system shown in FIG. Figure 1 As shown, the right side of the computer 11 is connected to the mobile phone 12, and the left side of the computer 11 is connected to the tablet 13. Fig.21 As shown, the computer 11 is connected to an external mouse 211 , and the mobile phone 12 is connected to an external mouse 212 .

[0460] like Fig.21 As shown in (a) of FIG. 1 , when user A manipulates the mouse 212 of the mobile phone 12 to move the mouse pointer 121 on the screen of the mobile phone 12 to the left, as shown in FIG. Fig.21 As shown in (a) and (b) of FIG. 1 , the keyboard and mouse can be crossed between the mobile phone 12 and the computer 11. In other words, the mouse pointer 121 corresponding to the mouse 212 of the mobile phone 12 can cross from the screen of the mobile phone 12 to the left to the screen of the computer 11. Fig.21 As shown in (b) in FIG. 1 , the mouse pointer 121 of the mobile phone 12 passes through the screen of the computer 11 and merges with the mouse pointer 111 of the mouse 211 of the computer 11 into a mouse pointer 131 , which is displayed in the form of an arrow.

[0461] User A can move the mouse pointer 131 on the screen of the computer 11 by operating the mouse 212 of the mobile phone 12 , and user B can move the mouse pointer 131 on the screen of the computer 11 by operating the mouse 211 of the computer 11 .

[0462] like Fig.21 As shown in (b) in FIG. 1 , when user A operates the mouse 212 of the mobile phone 12, user B operates the mouse 211 of the computer 11. In this case, an operation conflict occurs. In other words, the mouse pointer 131 on the screen of the computer 11 does not know where to move.

[0463] In other scenarios, such as Fig.21 As shown in (c), when the mouse pointer 121 of the mobile phone 12 has crossed to the screen of the computer 11, if user B operates the mouse 211 of the computer 11 to make the mouse pointer 111 cross from the screen of the computer 11 to the screen of the mobile phone 12. In other words, the mouse pointer 111 of the mouse 211 of the computer 11 crosses to the screen of the mobile phone 12, and the mouse pointer 121 of the mouse 212 of the mobile phone 12 crosses to the screen of the computer 11. In this case, since the mouse 212 of the mobile phone 12 has crossed to the computer 11, the mobile phone 12 has intercepted the keyboard and mouse events on the mobile phone 12, and when the mouse 211 of the computer 11 crosses to the mobile phone 12, the mobile phone 12 needs to stop intercepting the keyboard and mouse events on the mobile phone 12. This may cause crossing confusion.

[0464] Based on this consideration, an embodiment of the present application provides a keyboard and mouse traversal method, which can effectively reduce the occurrence of traversal confusion during the keyboard and mouse traversal process, and is conducive to providing the reliability of keyboard and mouse traversal.

[0465] There are two situations in which the keyboard and mouse crossover conflict occurs: In the first situation, the current mouse pointer is on the Android device, in other words, the Android device handles the keyboard and mouse crossover conflict; in the second situation, the current mouse pointer is on the PC, in other words, the PC handles the keyboard and mouse crossover conflict. The following introduces the keyboard and mouse crossover methods in the two situations respectively.

[0466] The following describes the keyboard and mouse crossover method in the first case (the Android device handles the keyboard and mouse crossover conflict). Fig. 22 , is a schematic diagram of the interaction flow of a keyboard and mouse traversal method provided in an embodiment of the present application. Fig. 22 The method described can be performed by an Android device. As an example and not limitation, Fig. 22 As shown, the keyboard and mouse traversal method may include the following steps:

[0467] S2201, the keyboard and mouse service module of Android device 1 monitors keyboard and mouse events.

[0468] The principle of this step is the same as that of S1214. For details, please refer to the description of S1214, which will not be repeated here.

[0469] S2202, the keyboard and mouse service module of Android device 1 determines whether the mouse corresponding to the monitored keyboard and mouse events is a local mouse.

[0470] If it is not the local mouse, continue to execute S2201.

[0471] If it is a local mouse, execute S2203.

[0472] The method for determining whether it is a local mouse is the same as the method in S1904. Please refer to the description in S1904 for details and will not be repeated here.

[0473] S2203, the keyboard and mouse service module of the Android device 1 determines whether the current mouse belongs to the PC.

[0474] In one implementation, S2202 and S2203 can be executed separately. In other words, first determine whether the mouse belongs to the local computer. If not, then determine which device the mouse belongs to. For example, by calling a preset function to determine whether it is a native keyboard and mouse or a virtual keyboard and mouse. The preset function can return a function value. When the function value is false, it indicates a virtual keyboard and mouse; when the function value is true, it indicates a native keyboard and mouse. If it is not a native keyboard and mouse, then obtain the device ID corresponding to the current mouse, and determine which device the current mouse belongs to based on the device ID.

[0475] In another implementation, S2202 and S2203 can be combined and executed. In other words, directly determine which device the mouse belongs to. For example, obtain the device ID corresponding to the current mouse, and determine which device the current mouse belongs to based on the device ID.

[0476] S2204, if the current mouse belongs to the PC, the keyboard and mouse service module of the Android device I sends a recovery instruction (first instruction) to the keyboard and mouse service module of the PC.

[0477] The restore command is used to inform the keyboard and mouse service module of the PC to restore the status.

[0478] S2205, after receiving the recovery instruction, the keyboard and mouse service module of the PC replies with a confirmation message to the keyboard and mouse service module of the Android device 1.

[0479] S2206, after receiving the resume instruction, the keyboard and mouse service module of the PC sends a stop instruction of intercepting events to the input framework of the PC.

[0480] S2207, after receiving the recovery instruction, the keyboard and mouse service module of the PC recovers the status.

[0481] S2206-S2207 and Fig.12 S1217-S1218 in the embodiment are the same, and can be seen in detail. Fig.12 Description in the example.

[0482] S2208, after the keyboard and mouse service module of Android device I receives the confirmation information replied by the keyboard and mouse service module of the PC, the keyboard and mouse service module of Android device I restores the status.

[0483] In this step, restoring the state includes stopping calling the preset interface of the input framework and restoring the mouse position.

[0484] S2208 and Fig.16 S1617-S1618 in the embodiment are the same, and can be seen in detail. Fig.16 Description in the Examples.

[0485] It should be noted that the above steps S2205-S2208 can be executed in parallel without any particular order.

[0486] S2209, if the current mouse belongs to Android device II, the keyboard and mouse service module of Android device I sends a recovery instruction (second instruction) to the keyboard and mouse service module of Android device II.

[0487] S2210, after receiving the recovery instruction, the keyboard and mouse service module of Android device II replies with confirmation information to the keyboard and mouse service module of Android device I.

[0488] S2211, after the keyboard and mouse service module of the Android device II receives the recovery instruction, the keyboard and mouse service module of the Android device II stops calling the preset interface of the input framework of the Android device II.

[0489] S2212, after receiving the restore instruction, the keyboard and mouse service module of the Android device II restores the state.

[0490] S2211-S2212 and Fig.16 S1617-S1618 in the embodiment are the same, and can be seen in detail. Fig.16 Description in the Examples.

[0491] It should be noted that S2211-S2212 can be executed in parallel without any particular order.

[0492] S2213, the keyboard and mouse module of Android device I sends a recovery instruction (third instruction) to the keyboard and mouse service module of the PC.

[0493] It should be noted that S2213 and S2209 can be executed in parallel without any particular order.

[0494] S2214, after receiving the recovery instruction, the keyboard and mouse service module of the PC updates the crossing status.

[0495] In this step, updating the crossing status refers to updating the crossing status from crossing to not crossing.

[0496] S2215, after the keyboard and mouse service module of Android device I receives the confirmation information replied by the keyboard and mouse service module of Android device II, the keyboard and mouse service module of Android device I restores the status.

[0497] S2215 is the same as S2208 and will not be described again here.

[0498] For example, in some application scenarios, Fig. 22 In the embodiment, Android device I may be the first device, and PC may be the second device; or, Android device I may be the first device, Android device II may be the second device, and PC may be the third device.

[0499] In the embodiment of the present application, when the keyboard and mouse are passed to the Android device and the user operates the mouse of the Android device, the Android device forcibly takes back the keyboard and mouse through the above method. In other words, as long as the local mouse is moved, the keyboard and mouse pass-through is stopped. This can avoid the occurrence of confusion in the pass-through.

[0500] The following describes the keyboard and mouse crossover method in the second case (the PC handles the keyboard and mouse crossover conflict). Fig.23 , is a schematic diagram of the interaction flow of a keyboard and mouse traversal method provided in an embodiment of the present application. Fig.23 The method described can be performed by a PC. As an example and not limitation, Fig.23 As shown, the keyboard and mouse traversal method may include the following steps:

[0501] S2301, the keyboard and mouse service module of the PC monitors keyboard and mouse events.

[0502] The principle of this step is the same as that of S2001. Please refer to the description of S2001 for details, which will not be repeated here.

[0503] S2302: The keyboard and mouse service module of the PC determines whether the mouse corresponding to the monitored keyboard and mouse event is a local mouse.

[0504] If it is not the local mouse, continue to execute S2301.

[0505] If it is a local mouse, execute S2303.

[0506] The method for determining whether it is a local mouse is the same as the method in S1904. Please refer to the description in S1904 for details and will not be repeated here.

[0507] S2303, the keyboard and mouse service module of the PC determines the Android device to which the current mouse belongs.

[0508] In one implementation, S2302 and S2303 can be executed separately. In other words, first determine whether the mouse belongs to the local device. If not, then determine which Android device the mouse belongs to. For example, by calling a preset function to determine whether it is a native keyboard and mouse or a virtual keyboard and mouse. The preset function can return a function value. When the function value is false, it indicates a virtual keyboard and mouse; when the function value is true, it indicates a native keyboard and mouse. If it is not a native keyboard and mouse, then obtain the device ID corresponding to the current mouse, and determine which device the current mouse belongs to based on the device ID.

[0509] In another implementation, S2302 and S2303 can be combined and executed. In other words, directly determine which device the mouse belongs to. For example, obtain the device ID corresponding to the current mouse, and determine which device the current mouse belongs to based on the device ID.

[0510] S2304, if the current mouse belongs to Android device I, the keyboard and mouse service module of the PC sends a recovery instruction (first instruction) to the keyboard and mouse service module of Android device I.

[0511] S2305, after receiving the recovery instruction, the keyboard and mouse service module of Android device I replies a confirmation message to the keyboard and mouse service module of the PC.

[0512] S2306, after the keyboard and mouse service module of Android device 1 receives the recovery instruction, the keyboard and mouse service module of Android device 1 stops calling the preset interface of the input framework of Android device 1.

[0513] S2307, after receiving the recovery instruction, the keyboard and mouse service module of Android device 1 recovers the status.

[0514] S2306-S2307 and Fig.16 S1617-S1618 in the embodiment are the same, and can be seen in detail. Fig.16Description in the Examples.

[0515] It should be noted that S2305-S2306 can be executed in parallel without any particular order.

[0516] S2308, after the keyboard and mouse service module of the PC receives the confirmation information replied by the keyboard and mouse service module of the Android device 1, the keyboard and mouse service module of the PC restores the status.

[0517] S2308 and Fig.12 S1218 in the embodiment is the same, for details, please refer to Fig.12 Description in the Examples.

[0518] It should be noted that the above Fig.23 In the embodiment, the keyboard and mouse belong to Android device I as an example for explanation. For the case where the keyboard and mouse belong to Android device II, the same as above Fig.23 The methods described in the embodiments are the same and will not be described again here.

[0519] For example, in some application scenarios, Fig.23 In the embodiment, the PC may be the first device, and the Android device 1 may be the second device.

[0520] In the embodiment of the present application, when the keyboard and mouse are passed to the Android device and the user operates the mouse of the Android device, the Android device forcibly takes back the keyboard and mouse through the above method. In other words, as long as the local mouse is moved, the keyboard and mouse pass-through is stopped. This can avoid the occurrence of confusion in the pass-through.

[0521] It should be noted that when the keyboard and mouse crossing is stopped, that is, each device restores the keyboard and mouse status, the mouse pointer of each device is restored to its own screen and displayed in its original form. Fig.24 , is a schematic diagram of a keyboard and mouse crossing forced recapture scenario provided in an embodiment of the present application. Fig.24 The scenario shown can be applied to Figure 1 In the communication system shown in FIG. Figure 1 As shown, the right side of the computer 11 is connected to the mobile phone 12, and the left side of the computer 11 is connected to the tablet 13. Fig.21 As shown, the computer 11 is connected to an external mouse 211 , and the mobile phone 12 is connected to an external mouse 212 .

[0522] like Fig.24 As shown in (a) of FIG. 1 , when user A manipulates the mouse 212 of the mobile phone 12 to move the mouse pointer 121 on the screen of the mobile phone 12 to the left, as shown in FIG. Fig.24 As shown in (a) and (b) of FIG. 1 , the keyboard and mouse can be crossed between the mobile phone 12 and the computer 11. In other words, the mouse pointer 121 corresponding to the mouse 212 of the mobile phone 12 can cross from the screen of the mobile phone 12 to the left to the screen of the computer 11. Fig.24As shown in (b) in FIG. 1 , the mouse pointer 121 of the mobile phone 12 passes through the screen of the computer 11 and merges with the mouse pointer 111 of the mouse 211 of the computer 11 into a mouse pointer 131 , which is displayed in the form of an arrow.

[0523] like Fig.24 As shown in (b) of FIG. 1 , when user B operates the mouse 211 of computer 11, computer 11 executes the above Fig.23 To stop the mouse and keyboard from crossing, use the method in Fig.24 As shown in (b) and (c), the mouse pointer 111 of the mouse 211 of the computer 11 returns to the center position on the screen of the computer 11 and is displayed in the form of an arrow; the mouse pointer 121 of the mouse 212 of the mobile phone 12 returns to the center position on the screen of the mobile phone 12 and is displayed in the form of a circle.

[0524] It should be noted that the above embodiments are all described with the PC as the communication center. In actual applications, an Android device can also be used as the center, that is, the Android device is connected to multiple terminal devices, and the keyboard and mouse connection is initiated by the Android device. In other words, the Android device is the initiator and the PC is the receiver. The keyboard and mouse traversal method with the Android device as the communication center is the same as the keyboard and mouse traversal method with the PC as the communication center in principle, and the embodiments of this application will not be repeated.

[0525] The various embodiments described herein may be independent solutions or may be combined according to internal logic, and all of these solutions fall within the protection scope of this application.

[0526] The above mainly describes the solution provided by the embodiment of the present application from the perspective of method steps. It is understandable that, in order to achieve the above functions, the electronic device implementing the method includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should be aware that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of protection of the present application.

[0527] The present application also provides a chip, which is coupled to a memory and is used to read and execute computer programs or instructions stored in the memory to execute the methods in the above embodiments.

[0528] The present application also provides an electronic device, which includes a chip, and the chip is used to read and execute a computer program or instruction stored in a memory, so that the method in each embodiment is executed.

[0529] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0530] This embodiment further provides a computer program product, wherein the computer-readable storage medium stores program codes. When the computer program product is run on a computer, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0531] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer execution instructions, and when the device is running, the processor can execute the computer execution instructions stored in the memory so that the chip executes the gaze point estimation method in the above-mentioned method embodiments.

[0532] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0533] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0534] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A keyboard and mouse traversal method, characterized in that: The method comprises: In a case where a first cursor of a first input device is displayed on a screen of a first device, the first device detects a first operation acting on a second input device, wherein the first device is connected to the second input device, the first device is not connected to the first input device, the first input device and the second input device are used to input control instructions, and the first operation is used to control a cursor movement displayed on the screen of the first device; In response to the first operation, the first device controls the first cursor to move out of the screen of the first device, and controls the second cursor of the second input device to be displayed on the screen of the first device.

2. The method according to claim 1, characterized in that The method further comprises: The first device detects a second operation, where the second operation is used to control a cursor movement displayed on a screen of the first device; In response to the second operation, the first device determines whether the input device corresponding to the second operation is the second input device; If the input device corresponding to the second operation is the second input device, the first device detects the first operation.

3. The method according to claim 1, characterized in that The first device controls the first cursor to move out of the screen of the first device, including: The first device detects a second device to which the input device corresponding to the first operation belongs; The first device sends a first instruction to the second device to instruct the second device to restore a cursor state; After receiving the first instruction, the second device controls the first cursor to move out of the screen of the first device and to be displayed on the screen of the second device.

4. The method according to claim 3, characterized in that The method further comprises: After receiving the first instruction, the second device replies with a first message to the first device to instruct the first device to restore the cursor state; After receiving the first information, the first device controls a second cursor of the second input device to be displayed on the screen of the first device.

5. The method according to claim 3, characterized in that The method further comprises: After the second device receives the first instruction, the second device stops intercepting the first input event corresponding to the third operation, wherein the third operation is used to control the movement of the first cursor of the first input device; In response to the third operation, the second device executes the control instruction corresponding to the first input event.

6. The method according to claim 3, characterized in that The method further comprises: After the second device receives the first instruction, the second device updates the crossing status of the second device to not crossing.

7. The method according to claim 4, characterized in that The method further comprises: After the first device receives the first information, the first device stops intercepting a second input event corresponding to a fourth operation, wherein the fourth operation is used to control movement of a second cursor of the second input device; In response to the fourth operation, the first device executes the control instruction corresponding to the second input event.

8. The method according to claim 4, characterized in that The method further comprises: After the first device receives the first information, the first device updates the crossing status of the first device to not crossing.

9. The method according to claim 3, characterized in that After the first device detects the second device to which the input device corresponding to the first operation belongs, the method further includes: The first device acquires device types of the first device and the second device, wherein the device types include a first type and a second type, and the first type of device is communicatively connected with each of the second type of devices respectively; If the device type of the first device is the first type and the device type of the second device is the second type, or the device type of the first device is the second type and the device type of the second device is the first type, the first device sends the first instruction to the second device to instruct the second device to restore the cursor state.

10. The method according to claim 9, characterized in that After the first device acquires the device types of the first device and the second device, the method further includes: If the device types of the first device and the second device are both the second type, the first device sends a second instruction to the second device to instruct the second device to restore the cursor state; The first device sends a third instruction to a third device to instruct the third device to update a traversal state, wherein the device type of the third device is the first type.

11. The method according to claim 10, characterized in that After the first device sends the second instruction to the third device, the method further includes: The third device receives the third instruction and updates the crossing status of the third device to not crossing.

12. The method according to any one of claims 9 to 11, characterized in that The first type is Windows system, and the second type is Android system.

13. The method according to any one of claims 3 to 12, characterized in that: The first device detecting the second device to which the input device corresponding to the first operation belongs includes: The first device acquires device information of an input device corresponding to the first operation; The first device determines, according to the device information, a second device to which the input device corresponding to the first operation belongs.

14. A communication system, characterized in that: The communication system includes a first device, a second device and a third device; The first device, the second device and the third device in the communication system are used to execute the method according to any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that: Used to store a computer program, which implements the method according to any one of claims 1 to 13 when executed by a processor.

16. A computer program product, characterized in that The computer program product includes a computer program code, and when the computer program code is executed on a computer, the computer is enabled to implement the method according to any one of claims 1 to 13.

Images