Key mouse traversal method, communication system and computer readable storage medium

Abstract

The application relates to the terminal control field, in particular to a keyboard mouse crossing method, a communication system and a computer readable storage medium. The method comprises the following steps: in the case that a cursor of an input device is displayed on a screen of a first device, in response to a first operation, the first device controls the cursor to move out of the screen of the first device and to be displayed on a screen of a second device, wherein the second device is located in a first direction of the first device and is connected with the first device; in response to a second operation, the second device controls the cursor to move out of the screen of the second device and to be displayed on a screen of a third device, wherein the third device is located in a second direction of the second device and is connected with the second device. According to the method, the input device can cross between different devices, the operation is simple, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of terminal control, and in particular to a keyboard and mouse traversal method, a communication system, and a computer-readable storage medium. Background Technology

[0002] With the development of terminal technology, the types and numbers of terminal devices are increasing. Currently, a single terminal device can no longer meet user needs, and users can use multiple terminal devices to meet their work and study requirements. However, when multiple terminal devices all require user processing, users need to switch between different terminal devices, which is cumbersome and detrimental to user experience. Summary of the Invention

[0003] This application provides a keyboard and mouse traversal method, a communication system, and a computer-readable storage medium, which can operate different terminals through a keyboard and mouse, and achieve seamless access between different terminal devices.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] Firstly, a keyboard and mouse method is provided, the method comprising:

[0006] When the cursor of the input device is displayed on the screen of the first device, the first device detects a first operation performed on the input device. The first operation is used to move the cursor displayed on the screen of the first device in a first direction, wherein the input device is used to input control commands.

[0007] In response to the first operation, the first device controls the cursor to move out of the screen of the first device and into the screen of the second device for display, wherein the second device is located in the first direction of the first device and establishes a connection with the first device;

[0008] The second device detects a second operation performed on the input device, the second operation being used to move a cursor displayed on the screen of the second device in a second direction;

[0009] In response to the second operation, the second device controls the cursor to move out of the screen of the second device and into the screen of the third device for display, wherein the third device is located in the second direction of the second device and is connected to the second device.

[0010] In this embodiment of the application, the input device may include a mouse, a keyboard, and a handwriting tablet, etc. For example, when the input device is a mouse, the cursor of the input device is the mouse pointer (or mouse cursor).

[0011] For example, the first device and the third device are devices running the Android system (Android devices), and the second device is a device running the Windows system (PC). The first device and the third device are respectively connected to the second device.

[0012] For example, in some application scenarios, the PC can be the second device, Android device I can be the first device, and Android device II can be the third device; or, the PC can be the second device, Android device II can be the first device, and Android device I can be the third device. In this application scenario, the second device communicates with both the first and third devices.

[0013] The keyboard and mouse traversal method provided in this application allows the input device to traverse between different devices. One set of input devices can control different devices without having to switch between different terminal devices, thus avoiding cumbersome operation and improving user experience.

[0014] In one implementation, the first direction and the second direction can include left or right. In another implementation, the first direction and the second direction can also include up or down.

[0015] In this embodiment of the application, the first operation and the second operation can be dragging or sliding the mouse.

[0016] In one implementation of the first aspect, in response to the first operation, the first device controls the cursor to move out of the screen of the first device and into the screen of the second device for display, including:

[0017] In response to the first operation, the first device locates a second device that has established a connection with the first device in the first direction;

[0018] The first device controls the cursor to move out of the screen of the first device and into the screen of the second device for display.

[0019] In this implementation, if the first device does not find a second device that has established a connection with the first device in the first direction, then keyboard and mouse traversal will not be performed.

[0020] In this embodiment, the cursor's movement direction (first direction) is the crossing direction, which is equivalent to determining whether there is a passable device in the crossing direction. Through the aforementioned pre-judgment step, the probability of keyboard and mouse crossing failure can be effectively reduced.

[0021] In one implementation of the first aspect, the method further includes:

[0022] In response to the first operation, the first device determines whether the input device is connected to the first device;

[0023] If the input device is connected to the first device, the first device updates the traversal status of the first device to "traversing".

[0024] The second device updates its traversal status to "traversing".

[0025] In this implementation, the input device is connected to the first device. The first operation is to move the cursor on the screen of the first device to the screen of the second device, which is equivalent to operating the input device of the first device, so that the input device of the first device can be transferred to the second device. By updating the transfer state of the first and second devices, the occurrence of transfer state confusion can be effectively reduced, thereby ensuring the reliability of keyboard and mouse transfer.

[0026] In one implementation of the first aspect, after the first device determines whether the input device is connected to the first device in response to the first operation, the method further includes:

[0027] If the input device is connected to the first device, the first device detects a third operation, the third operation being used to input the control command;

[0028] The first device intercepts the first input event corresponding to the third operation;

[0029] The first device sends the first input event to the second device;

[0030] After receiving the first input event, the second device executes the control command corresponding to the first input event.

[0031] In one implementation, when the first device is an Android device, the keyboard and mouse business module in the first device can call the preset interface of the input framework to intercept the first input event corresponding to the third operation.

[0032] In this embodiment, when the keyboard and mouse input from the first device is transferred to the second device, the first device intercepts the input events corresponding to the keyboard and mouse operations and sends the input events to the second device. The second device then executes the control commands corresponding to the operations performed on the input device. This allows different devices to share the input device without switching between different operation modes, avoiding cumbersome operations and improving the user experience. Furthermore, by intercepting the input events, the first device is prevented from responding to the keyboard and mouse operations, thus ensuring the reliability of the keyboard and mouse transfer.

[0033] In one implementation of the first aspect, in response to the second operation, the second device controls the cursor to move out of the screen of the second device and into the screen of the third device for display, including:

[0034] In response to the second operation, the second device locates a third device that has established a connection with the second device in the second direction;

[0035] The second device controls the cursor to move out of the screen of the second device and into the screen of the third device for display.

[0036] In this implementation, if the second device does not find a third device that has established a connection with the second device in the second direction, then keyboard and mouse traversal will not be performed.

[0037] In this embodiment, the cursor's movement direction (second direction) is the crossing direction, which is equivalent to determining whether there is a passable device in the crossing direction. Through the aforementioned pre-judgment step, the probability of keyboard and mouse crossing failure can be effectively reduced.

[0038] In one implementation of the first aspect, after the first device determines whether the input device is connected to the first device in response to the first operation, the method further includes:

[0039] If the input device is connected to the first device, in response to the fourth operation, the second device determines whether the first device and the third device are the same, and the fourth operation is used to move the cursor displayed on the screen of the second device in the second direction;

[0040] If the first device is the same as the third device, the first device updates the crossing status of the first device to "not crossed";

[0041] The second device updates its traversal status to "not traversed".

[0042] For example, the first device and the third device are the same, and the corresponding application scenario is that the keyboard and mouse of the first device travel to the second device, and then travel back to the first device from the second device.

[0043] In this embodiment of the application, by updating the crossing status of the first device and the second device in a timely manner, the occurrence of crossing confusion can be effectively reduced, which helps to improve the reliability of keyboard and mouse crossing.

[0044] In one implementation of the first aspect, after the second device determines whether the first device and the third device are the same in response to the fourth operation, the method further includes:

[0045] If the first device is the same as the third device, the first device stops intercepting the first input event corresponding to the third operation;

[0046] The first device executes the control command corresponding to the first input event.

[0047] In this embodiment, when the keyboard and mouse input from the first device moves to the second device and then back to the first device, the first device stops intercepting input events corresponding to the keyboard and mouse operations and executes control commands corresponding to the input device's operations. This allows for flexible switching of the input device between different devices, avoiding cumbersome operations and improving the user experience. Furthermore, by stopping the interception of input events, the first device's response to keyboard and mouse operations is ensured promptly, thereby guaranteeing the reliability of the keyboard and mouse transition.

[0048] In one implementation of the first aspect, after the second device determines whether the first device and the third device are the same in response to the fourth operation, the method further includes:

[0049] If the first device is the same as the third device, the first device controls the cursor to be displayed at a first preset position on the screen of the first device.

[0050] In this embodiment, when the keyboard and mouse of the first device travel to the second device and then back to the first device, the cursor returns to the first preset position on the screen of the first device, which helps the user quickly find the position of the cursor and thus improves the user experience.

[0051] In one implementation of the first aspect, after the second device determines whether the first device and the third device are the same in response to the fourth operation, the method further includes:

[0052] If the first device is different from the third device, the second device sends a first instruction to the first device;

[0053] After receiving the first instruction, the first device controls the cursor to move out of the screen of the second device and into the screen of the third device for display.

[0054] In this embodiment, the first instruction can be called a transfer instruction. This transfer instruction is used to inform the receiving end of the source keyboard and mouse event that a change has occurred.

[0055] In this embodiment, the first instruction is used to inform the first device receiving end that a transfer has occurred, thereby enabling the first device to respond promptly and control the cursor to move to the updated receiving end. This method improves the efficiency and reliability of keyboard and mouse movement.

[0056] In one implementation of the first aspect, after receiving the first instruction, the first device controls the cursor to move out of the screen of the second device and onto the screen of the third device for display, including:

[0057] After receiving the first instruction, the first device establishes a connection with the third device;

[0058] After the first device establishes a connection with the third device, the first device controls the cursor to move out of the screen of the second device and into the screen of the third device for display.

[0059] In this embodiment of the application, when the receiving end is transferred, the first device establishes a connection with the updated receiving end (the third device), thereby avoiding the first device communicating with the second device first and then the second device forwarding the communication to the third device, which effectively simplifies the communication process and helps to improve the keyboard and mouse response speed.

[0060] In one implementation of the first aspect, after the first device establishes a connection with the third device, the method further includes:

[0061] The second device updates its crossing status to "not crossed";

[0062] The third device updates its traversal status to "traversing".

[0063] In this implementation, updating the traversal state of the second and third devices can effectively reduce the occurrence of traversal state chaos, thereby ensuring the reliability of keyboard and mouse traversal.

[0064] In one implementation of the first aspect, after the first device establishes a connection with the third device, the method further includes:

[0065] If the first device loses connection with the third device, the first device controls the cursor to be displayed at a first preset position on the screen of the first device;

[0066] The first device maintains a communication connection with the second device.

[0067] Optionally, if the first device loses connection with the third device, the first device restores the keyboard and mouse positions.

[0068] One way to restore the keyboard and mouse position is to restore the keyboard and mouse position on the Android device based on the position of the keyboard and mouse when they move from the PC back to the screen of the Android device.

[0069] Another way to restore the keyboard and mouse position is to restore the keyboard and mouse position on the Android device based on the position of the keyboard and mouse as they move from the Android device to the PC screen.

[0070] Another way to restore the keyboard and mouse positions is to restore the Android device's keyboard and mouse positions to a preset position. For example, the preset position is the center of the Android device's screen.

[0071] In this embodiment, when the keyboard and mouse of the first device are moved to the third device and the connection between the first device and the third device is lost, the cursor is restored to the first preset position on the screen of the first device, which helps the user to quickly find the position of the cursor and thus improves the user experience.

[0072] In one implementation of the first aspect, after the first device establishes a connection with the third device, the method further includes:

[0073] If the first device loses connection with the third device, the first device stops intercepting the input events corresponding to the third operation;

[0074] The first device executes the control command corresponding to the input event.

[0075] In this embodiment, when the keyboard and mouse input from the first device is transferred to the third device and the connection between the first and third devices is lost, the first device stops intercepting input events corresponding to the keyboard and mouse operations and executes the control commands corresponding to the input device's operations. This allows for flexible switching of the input device between different devices, avoiding cumbersome operations and improving the user experience. Furthermore, by stopping the interception of input events, the first device's response to keyboard and mouse operations is ensured in a timely manner, thereby guaranteeing the reliability of the keyboard and mouse transfer.

[0076] In one implementation of the first aspect, after the first device determines whether the input device is connected to the first device in response to the first operation, the method further includes:

[0077] If the input device is not connected to the first device, the first device updates the traversal status of the first device to "not traversed".

[0078] For example, the corresponding application scenario could be that the cursor of the keyboard and mouse on another device is displayed on the screen of the first device, and then travels from the first device to the second device.

[0079] In this embodiment of the application, by updating the traversal status of the first device in a timely manner, the occurrence of traversal chaos can be effectively reduced, which helps to improve the reliability of keyboard and mouse traversal.

[0080] In one implementation of the first aspect, after the first device determines whether the input device is connected to the first device in response to the first operation, the method further includes:

[0081] If the input device is not connected to the first device, the first device sends a second instruction to the second device;

[0082] After receiving the second instruction, the second device determines whether the input device is connected to the second device;

[0083] If the input device is connected to the second device, the second device controls the cursor to move out of the screen of the first device and display it at a second preset position on the screen of the second device.

[0084] In this embodiment, the second instruction can be referred to as a return instruction. This return instruction is used to instruct the second device that the mouse pointer should return to the second device.

[0085] For example, the corresponding application scenario could be that the cursor of the keyboard and mouse on the second device is displayed on the screen of the first device, and then travels from the first device back to the second device.

[0086] In this embodiment, the cursor is restored to a second preset position on the screen of the second device, which helps the user quickly find the position of the cursor, thereby improving the user experience.

[0087] In one implementation of the first aspect, after determining whether the input device is connected to the second device, the method further includes:

[0088] If the input device is connected to the second device, the second device detects a fifth operation, which is used to input the control command.

[0089] The second device stops intercepting the second input event corresponding to the fifth operation;

[0090] The second device executes the control command corresponding to the second input event.

[0091] For example, the corresponding application scenario could be that the cursor of the keyboard and mouse on the second device is displayed on the screen of the first device, and then travels from the first device back to the second device.

[0092] In this embodiment, the second device stops intercepting input events corresponding to keyboard and mouse operations and executes control commands corresponding to the input device's operations. This allows for flexible switching between different devices, avoiding cumbersome operations and improving user experience. Furthermore, by stopping the interception of input events, the second device's timely response to keyboard and mouse operations is ensured, thereby guaranteeing the reliability of keyboard and mouse traversal.

[0093] In one implementation of the first aspect, after determining whether the input device is connected to the second device, the method further includes:

[0094] If the input device is connected to the second device, the second device updates the traversal status of the second device to "not traversed".

[0095] For example, the corresponding application scenario could be that the cursor of the keyboard and mouse on the second device is displayed on the screen of the first device, and then travels from the first device back to the second device.

[0096] In this embodiment of the application, by updating the traversal status of the second device in a timely manner, the occurrence of traversal chaos can be effectively reduced, which helps to improve the reliability of keyboard and mouse traversal.

[0097] In one implementation of the first aspect, after determining whether the input device is connected to the second device, the method further includes:

[0098] If the input device is connected to the second device, in response to the sixth operation, the second device determines a first crossing position on the screen of the second device, and the sixth operation is used to move the cursor displayed on the screen of the second device in the second direction;

[0099] The second device sends the first crossing location to the third device;

[0100] The third device determines the second crossing position on its screen based on the first crossing position;

[0101] The second device controls the cursor to move off the screen of the second device from the first crossing position;

[0102] The third device controls the cursor to enter the screen of the third device from the second crossing position.

[0103] For example, a corresponding application scenario could be that the cursor of the keyboard and mouse on the second device is displayed on the screen of the first device, and then travels from the first device back to the second device.

[0104] For example, if the mouse pointer's crossing position on the PC screen is (x1, y1), and the crossing direction is left or right, the PC's conversion module can be based on s y Change the transformation of y1, i.e., y1×s y The horizontal axis value is a preset value, which is either the minimum or maximum value of the screen's side length along the X-axis. R 1y R is the side length of the Android device's screen along the y-axis. 2y This represents the side length of the PC screen along the y-axis.

[0105] If the mouse cursor's position on the PC screen is (x1, y1), and the movement direction is upward or downward, the PC's conversion module can be based on s x Change the transformation of x1, i.e., x1×s x The value of the vertical axis is a preset value, which is either the minimum or maximum value of the screen's side length on the X-axis. R 1x R is the side length of the Android device's screen along the x-axis. 2x This represents the side length of the PC screen along the x-axis.

[0106] Using the above method, even when the resolutions of the second and third devices are different, the crossing position can be accurately determined, and the reliability of keyboard and mouse crossing is improved by increasing the orderliness of the crossing.

[0107] In one implementation of the first aspect, after responding to the sixth operation, the method further includes:

[0108] The second device updates its traversal status to "traversing".

[0109] By updating the traversal status of the second device in a timely manner, traversal chaos can be effectively reduced, thus improving the reliability of keyboard and mouse traversal.

[0110] In one implementation of the first aspect, after responding to the sixth operation, the method further includes:

[0111] The second device detects the seventh operation, which is used to input the control command;

[0112] The second device intercepts the third input event corresponding to the seventh operation;

[0113] The second device sends the third input event to the third device;

[0114] After receiving the third input event, the third device executes the control command corresponding to the third input event.

[0115] For example, a corresponding application scenario could be that the keyboard and mouse from the second device are transferred to the third device.

[0116] In one implementation, when the second device is a device running a Windows system, the second device can intercept the third input event corresponding to the seventh operation by intercepting events. These intercepted events are used to intercept keyboard and mouse events. The indication information of the intercepted events is used to instruct the input frame to set keyboard hooks, mouse hooks, and shortcut hooks. A hook is a platform in the Windows message handling mechanism where applications can set subroutines to monitor certain messages of a specified window, and the monitored window can be created by other processes. When a message arrives, it is processed before the target window's processing function. The hook mechanism allows applications to intercept and process window messages or specific events. A hook is actually a message-processing program segment that is hooked into the system via a system call. Whenever a specific message is sent, the hook program captures the message before it reaches the destination window; that is, the hook function gains control first. At this point, the hook function can process (change) the message, continue to pass the message without processing it, or forcibly terminate the message transmission.

[0117] Keyboard hooks are used to capture user input via the keyboard, such as capturing the user typing "memo information". Mouse hooks are used to capture user control actions via the mouse, such as clicking or dragging. Shortcut key hooks are used to capture user shortcut key actions, such as capturing Ctrl+C, Ctrl+V, Alt+TAB, or Ctrl+Alt+...

[0118] The DELETE operation.

[0119] By intercepting events, the mouse cursor travels from the PC to the Android device. The Windows system's input framework intercepts and listens for keyboard and mouse events, causing the business logic on the PC to not respond to these events, thus preventing the PC from affecting mouse events.

[0120] The second device intercepts input events corresponding to keyboard and mouse operations and sends them to a third device. The third device then executes the control commands corresponding to the operations performed on the input device. This allows different devices to share the input device without switching between different operation modes, avoiding cumbersome operations and improving the user experience. Furthermore, by intercepting input events, the second device is prevented from responding to keyboard and mouse operations, thus ensuring the reliability of keyboard and mouse crossover.

[0121] In one implementation of the first aspect, after determining whether the input device is connected to the second device, the method further includes:

[0122] If the input device is not connected to the second device, the second device determines a fourth device connected to the input device;

[0123] The second device sends a third instruction to the fourth device;

[0124] After receiving the third instruction, the fourth device controls the cursor to move out of the screen of the first device and into the screen of the second device.

[0125] In this embodiment, the third instruction can be referred to as a switching notification. This switching notification is used to inform the fourth device (keyboard and mouse) that it has been moved from the first device to the second device.

[0126] In one implementation, the second device sending a third instruction to the fourth device includes: the second device establishing a communication connection with the fourth device; and after the second device establishes a communication connection with the fourth device, the second device sending a third instruction to the fourth device.

[0127] In some application scenarios, the fourth device can be a third device. For example, a corresponding application scenario is that the keyboard and mouse of the third device travel to the first device, then from the first device to the second device, and finally from the second device back to the third device.

[0128] In this embodiment, the second device can determine the device to which the input device belongs (i.e., the source end), and then send a third instruction to the source end to instruct it to transfer the relevant data for keyboard and mouse traversal to the second device. Through this method, different devices can share the input device without switching between different operating modes, avoiding cumbersome operations, ensuring the reliability of keyboard and mouse traversal, and improving the user experience.

[0129] In one implementation of the first aspect, in response to the first operation, the first device controls the cursor to move out of the screen of the first device and into the screen of the second device for display, including:

[0130] In response to the first operation, the first device determines whether the cursor on the screen of the first device has reached the edge of the screen;

[0131] If the cursor on the screen of the first device reaches the edge of the screen, the first device controls the cursor to move out of the screen of the first device and into the screen of the second device for display.

[0132] In one implementation, determining whether the cursor has reached the edge of the screen includes: obtaining the current position of the cursor; obtaining the displacement of the cursor; calculating the position of the cursor after movement based on the current position and the displacement of the cursor; determining whether the position of the cursor after movement exceeds the screen; if the position of the cursor after movement exceeds the screen, then determining that the cursor has reached the edge of the screen.

[0133] It should be noted that if the first device determines whether the cursor has reached the edge of the screen, the above determination process is performed by the first device. If the second device determines whether the cursor has reached the edge of the screen, the above determination process is performed by the second device.

[0134] In a second aspect, a communication system is provided, the communication system comprising a first device, a second device, and a third device;

[0135] The second device is connected to both the first device and the third device.

[0136] The first device, the second device, and the third device in the communication system are used to perform the method as described in any of the first aspects.

[0137] Thirdly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in any one of the first aspects.

[0138] Fourthly, a computer program product is provided, the computer program product including computer program code, which, when run on a computer, causes the computer to perform the method as described in any one of the first aspects. Attached Figure Description

[0139] Figure 1 This is a schematic diagram of the communication system provided in an embodiment of this application;

[0140] Figure 2 This is a schematic diagram of a software architecture of a Windows system provided in an embodiment of this application;

[0141] Figure 3 This is a schematic diagram of a software architecture of an Android system provided in an embodiment of this application;

[0142] Figure 4 This is an interactive schematic diagram of establishing a keyboard and mouse connection provided in an embodiment of this application;

[0143] Figure 5 This is a schematic diagram of the interface of a PC manager provided in an embodiment of this application;

[0144] Figure 6This is a location diagram provided in an embodiment of this application;

[0145] Figure 7 This is a location diagram provided in another embodiment of this application;

[0146] Figure 8 This is a schematic diagram of a keyboard and mouse crossing method provided in an embodiment of this application;

[0147] Figure 9 This is a schematic diagram of a keyboard and mouse traversing a scene provided in an embodiment of this application;

[0148] Figure 10 This is a schematic diagram of the interaction scenario provided in the embodiments of this application;

[0149] Figure 11 This is a schematic diagram illustrating the implementation principle of a traversal method provided in an embodiment of this application;

[0150] Figure 12 This is a schematic diagram of the interaction flow of a keyboard and mouse traversal method provided in an embodiment of this application;

[0151] Figure 13 This is a schematic diagram of a screen crossing position provided in an embodiment of this application;

[0152] Figure 14 This is a schematic diagram of another keyboard and mouse traversal scenario provided in an embodiment of this application;

[0153] Figure 15 This is a schematic diagram illustrating the implementation principle of a traversal method provided in an embodiment of this application;

[0154] Figure 16 This is a schematic diagram of the interaction flow of a keyboard and mouse traversal method provided in an embodiment of this application;

[0155] Figure 17 This is a schematic diagram of a keyboard and mouse traversing a scene provided in an embodiment of this application;

[0156] Figure 18 This is a schematic diagram of a keyboard and mouse traversing a scene provided in an embodiment of this application;

[0157] Figure 19 This is a schematic diagram of the interaction flow of a keyboard and mouse traversal method provided in an embodiment of this application;

[0158] Figure 20 This is a schematic diagram of the interaction flow of a keyboard and mouse traversal method provided in an embodiment of this application. Detailed Implementation

[0159] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limiting purposes, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details.

[0160] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0161] It should also be understood that in the embodiments of this application, "one or more" refers to one, two, or more; "and / or" describes the relationship between the associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0162] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," "fourth," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0163] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0164] With the development of terminal technology, the types and numbers of terminal devices are increasing. Currently, a single terminal device can no longer meet user needs, and users can use multiple terminal devices to meet their work and study requirements. However, when multiple terminal devices all require user processing, users need to switch between different terminal devices, which is cumbersome and detrimental to user experience.

[0165] This application provides a keyboard and mouse traversal method, which allows operation of different terminal devices through a single input device, achieving seamless access between different terminal devices.

[0166] The method described in this application can be applied to terminal devices with screens and input access capabilities. For example, the terminal device in this application can be a personal computer (PC) (such as a desktop computer, laptop computer, mini-laptop, or ultrabook), a tablet computer, or a mobile phone. This application does not impose any special limitations on the specific form of the terminal device. For ease of explanation, the following embodiments use PCs, tablets, and mobile phones as examples.

[0167] In this embodiment, the input device can be a keyboard, mouse, or handwriting tablet (which may include a stylus), etc., used to input information to a terminal device. This embodiment does not impose any special limitations on the specific form of the input device. For ease of explanation, the keyboard and mouse will be referred to simply as keyboard and mouse in the following embodiments, and the keyboard and mouse will be used as an example for description.

[0168] In this embodiment, keyboard and mouse traversal can be achieved between two or more terminal devices. The following description uses three terminal devices as an example.

[0169] See Figure 1 This is a schematic diagram of the communication system provided in an embodiment of this application. Figure 1 As shown, the communication system includes a computer 11, a mobile phone 12, and a tablet 13. The computer 11 is communicatively connected to the mobile phone 12 and the tablet 13.

[0170] Figure 1 In the communication system shown, keyboard and mouse can be used interchangeably between computer 11 and mobile phone 12, between computer 11 and tablet 13, and between mobile phone 12 and tablet 13 via computer 11. In other words, computer 11, mobile phone 12, and tablet 13 can share the same mouse and keyboard.

[0171] Understandably, computer 11 can also achieve keyboard and mouse crossover with mobile phone 12 or tablet 13. In other words, computer 11 and mobile phone 12 can share the mouse and keyboard, or computer 11 and tablet 13 can share the mouse and keyboard.

[0172] In this embodiment, the keyboard and mouse of the computer 11 can be a keyboard and mouse connected to the computer 11, and the keyboard and mouse of the mobile phone 12 or tablet 13 can be a keyboard and mouse external to the mobile phone 12 or tablet 13.

[0173] The left or right side of computer 11 can be used to switch between keyboard and mouse with a single device. For example, computer 11 can support the mouse moving from the left side of the screen to tablet 13, or from the right side to mobile phone 12. Computer 11 can switch between keyboard and mouse with one device (mobile phone 12 or tablet 13) at a time. In other words, if a device is currently in a switching state, it will not switch between keyboard and mouse with other devices.

[0174] In this embodiment of the application, the software system of the terminal device involving keyboard and mouse traversal can be a Windows system or an Android system. For example, Figure 1 The computer 11 shown in the diagram uses the Windows operating system, while the mobile phone 12 and tablet 13 use the Android operating system. Devices running the Android system can be referred to as Android devices.

[0175] To better understand the embodiments of this application, the software architecture of the Windows system and Android system of the embodiments of this application is described below.

[0176] For example, see Figure 2 This is a schematic diagram of a Windows system software architecture provided in an embodiment of this application. Figure 2 As shown, the software architecture of a Windows system can include a driver layer, a basic framework layer, and an application layer.

[0177] The driver layer includes a virtual driver module and an input driver module. The virtual driver module sets up virtual drivers for the input devices of the Android device. The input driver module provides drivers for human interface devices (HIDs) to drive input devices connected to the PC (such as keyboards and mice).

[0178] The basic framework layer includes an input framework module, which reports input events to various business modules in the application layer, enabling these modules to perform corresponding functions. For example, the input framework module reports keyboard and mouse events to the keyboard and mouse business module in the application layer, allowing the module to process these events and thus achieve keyboard and mouse interaction.

[0179] The application layer includes a communication module (such as Magic Link) and a keyboard and mouse service module. The communication module provides a connection channel and data transmission capability between devices. The keyboard and mouse service module provides keyboard and mouse traversal functionality. This module includes a connection scheduling submodule, a keyboard and mouse management submodule, and a virtual driver loading submodule. The connection scheduling submodule provides a connection channel and data transmission capability with the communication module. The keyboard and mouse management submodule calls the connection scheduling submodule and the virtual driver loading submodule and provides the ability to handle keyboard and mouse events. The virtual driver loading submodule injects the keyboard and mouse events processed by the keyboard and mouse management submodule into the virtual driver module in the driver layer, so that the virtual driver module can report keyboard and mouse events to the input framework module in the system's base layer.

[0180] It should be noted that the communication module is used for communication between different devices. Data received by other modules must first be transmitted to the communication module before being transmitted to the other devices. In subsequent embodiments, to simplify the process, situations may arise where other modules communicate across devices besides the communication module. This can be understood as the other modules first transmitting data to the communication module, which then transmits it to the communication modules of the other devices, and finally, the communication modules of the other devices transmit it to the module that receives the required data.

[0181] It should be noted that data interaction between the keyboard and mouse service module and the communication module requires passing through the connection scheduling submodule. For example, data packets of keyboard and mouse events sent by other devices are first transmitted to the communication module, which then sends them to the connection scheduling subunit, and finally the connection scheduling subunit sends them to the keyboard and mouse management submodule. In subsequent embodiments, to simplify the process, if other submodules in the keyboard and mouse service module, other than the connection scheduling submodule, communicate with the communication module, it can be understood that the other submodules first transmit data to the connection scheduling submodule, which then transmits it to the communication module; or, the communication module first transmits data to the connection scheduling submodule, which then transmits it to the other submodules.

[0182] For example, see Figure 3 This is a schematic diagram of a software architecture for an Android system provided in an embodiment of this application. Figure 3 As shown, the software architecture of the Android system can include a driver layer, a basic framework layer, and an application layer.

[0183] The driver layer includes a virtual input driver module (Uinput Driver). This virtual input driver module is used to drive the input devices of Android devices.

[0184] The basic framework layer includes an input framework module, which reports input events to various business modules in the application layer, enabling these modules to perform corresponding functions. For example, the input framework module reports keyboard and mouse events to the keyboard and mouse business module in the application layer, allowing the module to process these events and thus achieve keyboard and mouse interaction.

[0185] The application layer includes a communication module (such as Magic Link) and a keyboard and mouse service module. The communication module provides a connection channel and data transmission capability between devices. The keyboard and mouse service module provides keyboard and mouse traversal functionality. This module includes a connection scheduling submodule and a keyboard and mouse management submodule. The connection scheduling submodule provides the connection channel and data transmission capability with the communication module. The keyboard and mouse management submodule calls the connection scheduling submodule, provides the ability to process keyboard and mouse events, and injects the processed events into the virtual input driver module in the driver layer. The virtual input driver module then reports the keyboard and mouse events to the input framework module in the system's base layer.

[0186] It should be noted that the communication module is used for communication between different devices. Data received by other modules must first be transmitted to the communication module before being transmitted to the other devices. In subsequent embodiments, to simplify the process, situations may arise where other modules communicate across devices besides the communication module. This can be understood as the other modules first transmitting data to the communication module, which then transmits it to the communication modules of the other devices, and finally, the communication modules of the other devices transmit it to the module that receives the required data.

[0187] It should be noted that data interaction between the keyboard and mouse service module and the communication module requires passing through the connection scheduling submodule. For example, data packets of keyboard and mouse events sent by other devices are first transmitted to the communication module, which then sends them to the connection scheduling subunit, and finally the connection scheduling subunit sends them to the keyboard and mouse management submodule. In subsequent embodiments, to simplify the process, if other submodules in the keyboard and mouse service module, other than the connection scheduling submodule, communicate with the communication module, it can be understood that the other submodules first transmit data to the connection scheduling submodule, which then transmits it to the communication module; or, the communication module first transmits data to the connection scheduling submodule, which then transmits it to the other submodules.

[0188] To better understand the embodiments of this application, the embodiments of this application are combined with the above. Figure 1 The communication system shown and Figure 2 , Figure 3 The software architecture shown provides a detailed description of the methods provided in the embodiments of this application.

[0189] In this embodiment, the condition for enabling keyboard and mouse traversal between terminal devices is that a communication connection has been established between the terminal devices. For example, a PC and an Android device can communicate via WiFi or Bluetooth, where WiFi can include a local area network or peer-to-peer (P2P); they can also communicate via wired connections, such as through a Type-A, Type-B, or Type-C data cable. In other implementations, the PC and Android device log in using the same account. For example, when computer 11, mobile phone 12, and tablet 13 log in using the same account, computer 11, mobile phone 12, and tablet 13 establish a communication connection.

[0190] For example, see Figure 4 This is an interactive schematic diagram illustrating the establishment of a keyboard and mouse connection provided in an embodiment of this application. For example... Figure 4 As shown, the process of establishing a keyboard and mouse connection between a PC and any Android device may include the following steps:

[0191] S401, the PC's keyboard and mouse management submodule is started.

[0192] The keyboard and mouse management submodule within the PC's keyboard and mouse service module can start automatically when the PC boots up, or it can be started manually by the user. For example, if started manually, the user can enable the keyboard and mouse traversal function through the PC's settings interface. In response to the user enabling the keyboard and mouse traversal function, the PC's keyboard and mouse service module starts, and correspondingly, the keyboard and mouse management submodule within that module starts.

[0193] S402, the PC's keyboard and mouse management submodule sends instruction information a to the PC's connection scheduling submodule.

[0194] Instruction information 'a' is used to instruct the PC's connection scheduling submodule to establish a communication connection with the Android device.

[0195] S403, after receiving the indication information a, the PC's connection scheduling submodule sends the request information b to the PC's communication module.

[0196] Request information b is used to request the PC's communication module to inform it of the detected online Android devices.

[0197] The PC's communication module can detect whether an Android device is online. If an Android device is online, based on request information b, the PC's communication module sends the device identifier (identity, ID) of the online Android device to the PC's keyboard and mouse management submodule through the PC's connection scheduling submodule. If no Android device is online, the PC's communication module can monitor whether an Android device is online in real time, or check whether an Android device is online at preset intervals.

[0198] The PC's connection scheduling submodule sends request information b to the PC's communication module. This can also be understood as the PC's connection scheduling submodule registering device online / offline fences on the PC's communication module. When the PC's communication module detects an online device, it returns the device ID to the PC's keyboard and mouse management submodule through the connection scheduling submodule.

[0199] S404, the keyboard and mouse management submodule of the Android device is launched.

[0200] The keyboard and mouse management submodule within the Android device's keyboard and mouse service module can start automatically when the Android device boots up, or it can be started manually by the user. For example, if started manually, the user can enable the keyboard and mouse traversal function through the settings interface on the Android device. In response to the user enabling the keyboard and mouse traversal function, the Android device's keyboard and mouse service module starts, and correspondingly, the keyboard and mouse management submodule within that module starts.

[0201] S405, after the PC's communication module receives the instruction information b, it discovers each other with the Android device's communication module, forming a trust loop.

[0202] As mentioned above, the prerequisite for keyboard and mouse crossover between a PC and an Android device is that a communication connection has been established between the two devices. This communication connection refers to the ability of the PC's communication module and the Android device's communication module to discover each other and establish a trust loop.

[0203] It should be noted that S402 is a step executed after the PC's keyboard and mouse management submodule starts. This embodiment does not limit the order in which steps S402 and S404 are executed. In other words, if step S404 has been executed before S402, meaning the Android device's keyboard and mouse management submodule has started, then after executing S402, the PC's communication module can detect the Android device's online status and thus execute S405. If step S404 has not been executed before S402, meaning the Android device's keyboard and mouse management submodule has not started, then after executing S402, the PC's communication module continuously checks for the presence of an online Android device, either in real-time or at preset intervals, until the Android device's keyboard and mouse management submodule starts and the PC's communication module detects the presence of an online Android device, at which point S405 is executed.

[0204] A trust loop represents a communication connection established between different terminal devices. For example, if a PC and an Android device are within the same Bluetooth range, connected to the same Wi-Fi network, connected via the same data cable, or logged into the same account, the communication modules of the PC and the Android device can sense and discover each other, forming a trust loop. The PC's communication module can then obtain the Android device's device ID.

[0205] The device ID can be at least one of numbers, letters, or symbols, and this application does not limit this.

[0206] There can be one or more Android devices online, and this application embodiment does not limit this. If there are multiple online Android devices, the PC's communication module can send the device IDs of the multiple Android devices to the PC's keyboard and mouse service module.

[0207] In one implementation, the PC's keyboard and mouse service module may include a device management submodule. When the PC first establishes a communication connection with an Android device, the PC's device management submodule can authenticate the Android device based on its device ID. Upon successful authentication, the PC's connection scheduling submodule notifies the PC's communication module to execute S405 and stores the device ID. During subsequent communication connections, when the PC's communication module detects an online Android device, it reports the Android device's device ID to the PC's device management submodule via the PC's connection scheduling submodule. The PC's device management submodule checks if the device ID is already stored. If it is, the PC's connection scheduling submodule sends an instruction to the PC's communication module, instructing the communication module to execute S405.

[0208] S406, after the trust loop is formed, the communication module of the Android device sends a notification c to the connection scheduling submodule of the Android device.

[0209] Notification 'c' is used to instruct the Android device's connection scheduling submodule to report to the Android device's keyboard and mouse management submodule that the communication connection with the PC has been successfully established.

[0210] S407, after receiving notification c, the connection scheduling submodule of the Android device sends notification d to the keyboard and mouse management submodule of the Android device.

[0211] Notification d is used to indicate that the communication connection with the PC has been successful.

[0212] S408, after the trust loop is formed, the PC's communication module sends a notification e to the PC's connection scheduling submodule.

[0213] Notification e is used to instruct the PC's connection scheduling submodule to report to the PC's keyboard and mouse management submodule that the communication connection with the Android device has been successfully established.

[0214] S409, after receiving notification e, the PC's connection scheduling submodule sends notification f to the PC's keyboard and mouse management submodule.

[0215] The notification f is used to indicate that the communication connection with the Android device has been successful.

[0216] It should be noted that steps S406 and S408 are not sequential and can be processed in parallel.

[0217] At this point, the keyboard and mouse connection between the PC and the Android device has been successfully established.

[0218] Understandably, to achieve keyboard and mouse crossover between a PC and an Android device, it's necessary not only to establish a communication connection between the PC and the Android device (i.e., forming a trust loop between the PC's communication module and the Android device's communication module), but also to establish a keyboard and mouse connection between the PC and the Android device (i.e., an event connection between the PC's keyboard and mouse service module and the Android device's keyboard and mouse service module). In other words, Figure 4 The diagram illustrates the process of establishing a keyboard and mouse connection between a PC and an Android device. A successful keyboard and mouse connection between the PC and the Android device indicates that the event connection between the keyboard and mouse service modules of the PC and the Android device has been successfully established.

[0219] It should be noted that, Figure 4 The embodiments illustrate the process of establishing a keyboard and mouse connection centered on a PC; in other words, the PC initiates the keyboard and mouse connection. In PC-centric application scenarios, to achieve keyboard and mouse connectivity between a PC and multiple Android devices, the PC needs to establish a keyboard and mouse connection with each Android device separately. In other embodiments, the keyboard and mouse connection can also be established centered on the Android device, i.e., the Android device initiates the keyboard and mouse connection. In Android device-centric application scenarios, to achieve keyboard and mouse connectivity between an Android device and multiple terminal devices, the Android device needs to establish a keyboard and mouse connection with each terminal device separately. For ease of explanation, the application scenarios involved in the embodiments of this application are all PC-centric scenarios.

[0220] Establishing a keyboard and mouse connection between a PC and an Android device can be done in two ways. One possible approach is automatic connection establishment. In this method, after the PC's keyboard and mouse management submodule in S401 starts, subsequent steps are executed automatically, and the PC can choose the connectable device. The other possible approach is manual connection establishment. In this method, after the PC's keyboard and mouse management submodule in S401 starts, when the PC's communication module detects an online Android device, the user can manually select the connectable device in a visual interface. After the user's selection, the subsequent steps in S401 are executed.

[0221] The keyboard and mouse connection settings may be located in different places on different devices. For example, the keyboard and mouse connection settings may be found in the PC Manager on PC 11. On mobile phones 12 or tablets 13, the keyboard and mouse connection settings may be found in the Control Center or Settings.

[0222] For example, see Figure 5 This is a schematic diagram of the interface of a computer manager provided in an embodiment of this application. Figure 5As shown, the PC Manager interface can include options such as Homepage, Multi-screen Collaboration, Smart Interconnection, Online Tips, Official Services, Smart Audio-Visual, and System Optimization. When PC 11 detects that the user clicks the Smart Interconnection option with the mouse, it can display the keyboard and mouse sharing interface. This interface includes two options. The first option is "Double-touch the mouse pointer twice on the side edge of the computer screen to establish a connection." The second option is "Automatically adjust the screen arrangement according to the device's placement."

[0223] Understandably, the first option is the one mentioned above that automatically establishes a keyboard and mouse connection. When this option is selected, computer 11 can automatically connect to devices that meet the keyboard and mouse traversal requirements. When the first option (i.e., the option to automatically establish a communication connection) is not selected, computer 11 provides a way to manually establish a keyboard and mouse connection to connect to devices that meet the keyboard and mouse traversal requirements.

[0224] As an example of manually establishing a keyboard and mouse connection, such as Figure 5 As shown, the first option is unselected; in this case, computer 11 provides a manual connection method. The keyboard and mouse sharing interface can display connected and online devices. Figure 5 In the keyboard and mouse sharing interface shown in (a), a mobile phone 12 is connected to the left side of computer 11, along with two other online devices (devices with established trust loops). These two online devices include tablet 13 and mobile phone 14. Users can select and connect to these two online devices by clicking or dragging. Figure 5 As shown in (a), the user controls the mouse pointer to click on the icon on tablet 13. In response to clicking the icon on tablet 13, computer 11 establishes a keyboard and mouse connection with tablet 13 and displays as shown in (a). Figure 5 The keyboard and mouse shared interface is shown in (b) above. Figure 5 As shown in (b) in the image, in the keyboard and mouse sharing interface, a tablet 13 is connected to the right side of the computer 11.

[0225] like Figure 5 In the keyboard and mouse sharing interface shown in (a) and (b), the icon of the connected device on computer 11 can display a selection indicator 501. The user can disconnect the PC from the device by manipulating the mouse to click the selection indicator 501 or drag the icon of the connected device. For example, the user can disconnect the PC from the device by manipulating the mouse to click the selection indicator 501 or drag the icon of the connected device. Figure 5 On the interface shown in (b), clicking the selection icon 501 on the left side of the mobile phone on the computer 11 will, in response to this user action, disconnect the keyboard and mouse connection between the computer 11 and the mobile phone 12, and display the following... Figure 5 The keyboard and mouse shared interface is shown in (c) above. Figure 5As shown in (c), in the keyboard and mouse sharing interface, no device is connected to the left side of computer 11, tablet 13 is connected to the right side of computer 11, and mobile phone 12 is displayed in the area where devices can be connected.

[0226] In the manual keyboard and mouse connection setup described above, the PC determines the device's trajectory direction based on the location of the device selected by the user in the keyboard and mouse sharing interface. For example, ... Figure 5 As shown in (b), the left side of computer 11 is mobile phone 12 and the right side is tablet 13. Therefore, computer 11 determines that the direction of mobile phone 12 is to the left (that is, when the mouse in computer 11 is moved to the left, the mouse pointer can reach the screen of mobile phone 12), and the direction of tablet 13 is to the right (that is, when the mouse in computer 11 is moved to the right, the mouse pointer can reach the screen of tablet 13).

[0227] As an example of automatically establishing a keyboard and mouse connection, if Figure 5 The first option is selected. When computer 11 detects that the mouse pointer touches the edge of the computer screen twice consecutively, computer 11 can automatically connect to devices that meet the keyboard and mouse crossover conditions (such as being connected to the same network and / or logged into the same account). For example, after executing S403 above, if the communication module of computer 11 detects an online Android device, then S405 is executed. If the communication module of computer 11 detects multiple online Android devices, it can establish a keyboard and mouse connection with each online Android device separately, or automatically select one or more devices to establish a keyboard and mouse connection.

[0228] In the automatic keyboard and mouse connection setup mode, the PC can automatically set the crossover direction of the connected devices, or the user can manually set the crossover direction of the connected devices.

[0229] As an example of automatically setting the crossing direction, when the user enables the option to automatically identify the direction, the PC can detect the location of the device with the established keyboard and mouse connection using ultrasonic methods, and automatically set the crossing direction of the device with the established keyboard and mouse connection based on the detected location.

[0230] For example, in Figure 5 In the interface shown, if the user checks or selects the option "Automatically adjust screen arrangement according to device placement," the PC detects this selection and, in response, can automatically identify the location of currently connected devices. For example, in... Figure 1In the application scenario shown, when the user enables the automatic orientation recognition option, the mobile phone 12 or tablet 13 can detect its own movement through the motion sensor. Once the movement stops, the computer 11 calculates its position relative to the mobile phone 12 or tablet 13 using data acquired by the ultrasonic sensor, and transmits this position to the computer 11. Upon receiving this position, the computer 11 determines that the mobile phone 12 is to the left of the computer 11 and the tablet 13 is to the right. Then, the computer 11 sets the direction of movement for the mobile phone 12 to the left (i.e., when the mouse cursor is moved to the left on the computer 11, it reaches the screen of the mobile phone 12), and sets the direction of movement for the tablet 13 to the right (i.e., when the mouse cursor is moved to the right on the computer 11, it reaches the screen of the tablet 13).

[0231] Understandably, if the location of a currently connected keyboard and mouse device changes when the user has the automatic orientation detection option enabled, the PC can detect the change in location of the connected device using ultrasonic waves and automatically adjust the direction of movement of the connected device.

[0232] It should be noted that the keyboard and mouse can not only move to the left or right of the screen, but also to the top or bottom of the screen. This application embodiment does not limit this.

[0233] In one possible implementation, the screen of the terminal device can be divided into left, right, and center regions. For example, computer 11 can establish a coordinate system with the center of the screen as the origin, and the parallel lines of the long and wide sides of the screen as the parallel lines to facilitate the division of the left, right, and center regions. If computer 11 detects that mobile phone 12 and / or tablet 13 are in the left region, it can determine that mobile phone 12 and / or tablet 13 are on the left side of computer 11. If computer 11 detects that mobile phone 12 and / or tablet 13 are in the right region, it can determine that mobile phone 12 and / or tablet 13 are on the right side of computer 11. If computer 11 detects that mobile phone 12 and / or tablet 13 are in the center region, it can determine that mobile phone 12 and / or tablet 13 are in the center of computer 11.

[0234] For example, see Figure 6 This is a location diagram provided in an embodiment of this application. For example... Figure 6 As shown, computer 11 defines the region between (-45°, 45°) in the coordinate system as the right region, the region between (45°, 135°) and (-45°, -135°) as the middle region, and the region between (-135°, 135°) as the left region. If the angle of mobile phone 12 relative to computer 11 is between (-45°, 45°), then computer 11 detects that mobile phone 12 is in the right region, and can determine that mobile phone 12 is to the right of computer 11.

[0235] In another possible implementation, the screen of the terminal device can be divided into left, right, top, and bottom sections. For example, see [link to example]. Figure 7 This is a location diagram provided in another embodiment of this application. For example... Figure 7 As shown, computer 11 defines the region between (-45°, 45°) in the coordinate system as the right region, the region between (45°, 135°) as the upper region, the region between (-45°, -135°) as the lower region, and the region between (-135°, 135°) as the left region. If the angle of mobile phone 12 relative to computer 11 is between (45°, 135°), then computer 11 detects that mobile phone 12 is in the upper region and can determine that mobile phone 12 is above computer 11.

[0236] It should be noted that, Figure 6 and Figure 7 The angle ranges for the area divisions shown are merely an example. The angle ranges for each area can be adjusted based on the user's experience. For example, the angle range of the middle area can be adjusted from (45°, 135°) to (30°, 150°) and from (-45°, -135°) to (-30°, -150°) based on the user's experience.

[0237] As an example of manually setting the traversal direction, when the PC detects the mouse pointer at the edge of the screen, a bubble can be displayed at the edge to indicate the presence of a traversable device (a device with an established keyboard and mouse connection). When the user interacts with the bubble and selects a traversable device, the PC sets the traversal direction of the selected device based on the current position of the mouse pointer on the edge of the screen.

[0238] For example, see Figure 8 This is a schematic diagram illustrating a keyboard and mouse traversal method provided in an embodiment of this application. Figure 8 As shown, the devices that can be traversed by computer 11 include mobile phone 12 and tablet 13. When computer 11 detects the mouse pointer at the right edge of the computer screen, a bubble 801 is displayed at the right edge of the computer screen to indicate the presence of traversable devices. The bubble 801 displays the icons 8011 of mobile phone 12 and 8012 of tablet 13.

[0239] Users can use the mouse to control the mouse pointer to click on the icon 8011 on the phone 12. In response to the user's operation, the computer 11 sets the direction of movement of the phone 12 to the right, that is, the mouse in the computer 11 is moved to the right, and the mouse pointer can reach the screen of the phone 12.

[0240] Users can also control the mouse pointer to click on the icon 8012 on the tablet 13 by operating the mouse. In response to the user's operation, the computer 11 sets the direction of movement of the tablet 13 to the right, that is, the mouse of the computer 11 is moved to the right, and the mouse can reach the screen of the tablet 13.

[0241] The bubble is merely an example of a name, and this application embodiment does not limit its usage. The number of traversable devices (devices that have formed a trust loop with the PC) can be one or more, and this application embodiment does not limit its usage. When there is only one traversable device, there can be only one bubble. The bubble can display the icon and model of the traversable device; for example, if the traversable device is a mobile phone, it can display the mobile phone icon and model, or it can not display the icon and / or model of the traversable device, and this application embodiment does not limit its usage.

[0242] When there are multiple traversable devices, the number of bubble boxes can be one or the same as the number of traversable devices; this embodiment does not limit this. If there are multiple traversable devices, there are also multiple bubble boxes. Each bubble box can correspond to one traversable device, and each bubble box can display the icon and / or model number of the corresponding traversable device. Users can click on a bubble box to select one traversable device from multiple traversable devices for keyboard and mouse traversal, or they can click on the icon or model number of a traversable device to select one for keyboard and mouse traversal; this embodiment does not limit this. If there are multiple traversable devices, there is only one bubble box, and this bubble box can display the icons and / or models of multiple traversable devices. Users can click on the icon or model number of a traversable device to select one for keyboard and mouse traversal.

[0243] Figure 8 This application only shows one example of a bubble. Examples of bubble displays will not be provided for cases where the bubble does not display the icon and / or model of the passable device, or where multiple bubble displays are present when there are multiple passable devices, or other display formats of bubble.

[0244] In one possible implementation, the computer 11 displays a bubble corresponding to the currently passable device each time it detects the mouse pointer at the edge of the computer screen.

[0245] In another possible implementation, when computer 11 first detects the mouse pointer on the edge (side1) of the computer screen, it displays a bubble corresponding to the currently accessible device. After the user selects a accessible device, if computer 11 detects the mouse pointer on the edge (side1) of the computer screen again, it stops displaying the bubble corresponding to the currently accessible device. However, when computer 11 detects the mouse pointer on the edge (side2) of the computer screen, it displays the bubble corresponding to the currently accessible device. In this implementation, when computer 11 detects the mouse pointer on the edge (side2) of the computer screen, it can display bubble boxes corresponding to all currently accessible devices, or it can only display bubble boxes corresponding to the currently unselected accessible devices. For example, the currently accessible devices are device 1, device 2, and device 3. If the mouse pointer of computer 11 is at the right edge of the computer screen and the user selects the bubble corresponding to device 1, it means that the user has set device 1 as the passing device to the right of computer 11. That is, if the mouse of computer 11 is moved to the right, the mouse can reach the screen of device 1. When the mouse pointer of computer 11 is at the left edge of the computer screen, computer 11 displays the bubble corresponding to device 2 and device 3 on the left edge of the computer screen for the user to select. If the user selects device 2, it means that the user has set device 2 as the passing device to the left of computer 11. That is, if the mouse of computer 11 is moved to the left, the mouse can reach the screen of device 2.

[0246] It should be noted that the direction of mouse movement can be related to or independent of the position between the devices. This embodiment does not impose such a limitation. For example, if the direction of mouse movement is related to the position between the devices, and device 1 is to the left of device 2, then moving the mouse on device 2 to the left will allow the mouse to reach the screen of device 1. If the direction of mouse movement is independent of the position between the devices, and device 1 is to the left of device 2, the direction of movement for device 1 can be set to the right of device 2 using the aforementioned manual communication establishment method or manual setting method. That is, moving the mouse on device 2 to the right will allow the mouse to reach the screen of device 1.

[0247] Once a keyboard and mouse connection is established between terminal devices, keyboard and mouse traversal between these devices can be achieved. The following section describes the PC-centric keyboard and mouse traversal method. In this PC-centric method, the PC typically initiates the keyboard and mouse connection, acting as the communication center. The PC can connect to one or more traversable Android devices; in other words, keyboard and mouse traversal between devices must pass through the PC. The PC-centric keyboard and mouse traversal method can be applied to… Figure 1 The communication system shown. For example... Figure 1 As shown, computer 11 is connected to mobile phone 12 and tablet 13 respectively. At this time, computer 11 is the communication center for mobile phone 12 and tablet 13.

[0248] In this embodiment of the application, in the PC-centric traversal method, when a keyboard and mouse traversal event is initiated by computer 11, and the keyboard and mouse of computer 11 traverse to mobile phone 12 or tablet 13, computer 11 can be referred to as the source device or initiator, and mobile phone 12 or tablet 13 can be referred to as the peer device or receiver. When a keyboard and mouse traversal event is initiated by mobile phone 12, and the keyboard and mouse of mobile phone 12 traverse to computer 11 or tablet 13, mobile phone 12 can be referred to as the source device or initiator, and computer 11 or tablet 13 can be referred to as the peer device or receiver. However, this embodiment of the application is not limited to this.

[0249] PC-centric keyboard and mouse navigation can be divided into two methods: In the first method, the main operation is on the PC side, but the Android device needs to be accessed. In this method, the PC's keyboard and mouse can be moved to the Android device. For example, Figure 1 In the first method, the keyboard and mouse of the Android device 11 can be transferred to the screen of the mobile phone 12 or tablet 13. In the second method, the operation is primarily performed on the Android device side, requiring interaction with the PC. In this method, the keyboard and mouse of the Android device can be transferred to the PC. For example, Figure 1 The keyboard and mouse of the mobile phone 12 or the keyboard and mouse of the tablet 13 are transferred to the screen of the computer 11.

[0250] The two methods of crossing the trestle will be described below.

[0251] As an example of the first method of crossover (crossover of PC keyboard and mouse to Android device), see Figure 9 This is a schematic diagram of a keyboard and mouse crossing scene provided in an embodiment of this application. Figure 9 The keyboard and mouse traversal scenario shown can be applied to Figure 1 In the communication system shown, such as Figure 1 As shown, the right side of computer 11 is connected to mobile phone 12, and the left side is connected to tablet 13.

[0252] like Figure 9 As shown in (a), when the user manipulates the mouse of computer 11 to move the mouse pointer 111 on the screen of computer 11 to the right, as... Figure 9 As shown in (a) and (b), computer 11 can achieve keyboard and mouse crossover with mobile phone 12. In other words, the mouse pointer 111 displayed on computer 11 can cross from the screen of computer 11 to the right onto the screen of mobile phone 12. When the mouse pointer 111 of computer 11 crosses the screen of mobile phone 12, as shown in (a) and (b), the mouse pointer 111 of computer 11 can cross the screen of mobile phone 12. Figure 9 As shown in (b) above, the mouse pointer 111 can be displayed as a circle on the screen of the mobile phone 12. Figure 9 As shown in (b), the user can also move the mouse pointer 111 on the mobile phone screen to the left by operating the mouse of the computer 11, such as... Figure 9As shown in (b) and (c), the mouse pointer 111 on the screen of mobile phone 12 moves to the left and crosses back onto the screen of computer 11, as shown in (b) and (c). Figure 9 As shown in (c), the mouse pointer 111 is displayed as an arrow on the screen of computer 11.

[0253] In some implementations, the mouse cursor on computer 11 can travel to different Android device screens, and the mouse pointer 111 can be displayed in the same way on different Android device screens. For example, Figure 1 In the communication system shown, computer 11 can also achieve keyboard and mouse crossover with tablet 13. In other words, the mouse pointer 111 displayed on computer 11 can cross from the screen of computer 11 to the screen of tablet 13 from the left. When the mouse pointer 111 of computer 11 crosses the screen of tablet 13, the mouse pointer 111 can also be displayed in the form of a circle on the screen of tablet 13.

[0254] In other implementations, the mouse cursor of computer 11 can travel across the screens of different Android devices, and the mouse pointer 111 can be displayed in different forms on the screens of different Android devices. For example, when the mouse pointer 111 of computer 11 travels across the screen of tablet 13, the mouse pointer 111 can be displayed in the form of a diamond on the screen of tablet 13.

[0255] In this example, the external mouse connected to computer 11 is displayed as an arrow on the computer 11 screen, and the external mouse connected to mobile phone 12 is displayed as a circle on the mobile phone 12 screen. The mouse pointer 111 moves from the computer 11 screen to the mobile phone 12 screen, where it is displayed as a circle; the mouse pointer 111 moves back from the mobile phone 12 screen to the computer 11 screen, where it reverts to an arrow. It should be noted that in other examples, the mouse pointer may display in other forms after moving from the computer to the mobile phone; this is not limited here.

[0256] After the keyboard and mouse of computer 11 are transferred to mobile phone 12 or tablet 13, users can use the mouse to click, drag or double-click on mobile phone 12 or tablet 13, and use the keyboard to input information on mobile phone 12 or tablet 13.

[0257] See Figure 10 This is a schematic diagram of an interactive scenario provided in an embodiment of this application. For example... Figure 10 In the example, after the mouse pointer 111 on computer 11 crosses the screen of mobile phone 12, the user can move the position of the mouse pointer 111 on the mobile phone screen by operating the mouse. Figure 10As shown in (a), the mouse pointer 111 is positioned over the "Notes" app icon. The user can click the "Notes" app icon using the mouse pointer 111, and the phone 12, in response to the user's click, can display the following... Figure 10 The interface shown in (b) is as follows. Figure 10 As shown in (b), the mobile phone 12 displays the memo interface. The user can enter the message "memo" in the memo interface using the keyboard, and the mobile phone 12 responds to the user's operation of entering the message in the memo interface by displaying the text "memo" in the memo interface.

[0258] To better understand the embodiments of this application, the following is combined with... Figure 2 and Figure 3 The software architecture shown illustrates the implementation principle of the first traversal method (the keyboard and mouse of the PC traversing to the Android device).

[0259] See Figure 11 This is a schematic diagram illustrating the implementation principle of a traversal method provided in an embodiment of this application. For example... Figure 11 As shown, the implementation principle of PC keyboard and mouse crossing over to Android devices can include the following steps:

[0260] Step 000: The Windows system's input driver detects the PC's keyboard and mouse events and reports these events to the Windows input framework.

[0261] Step 001: The input framework of the Windows system reports the received keyboard and mouse events to the keyboard and mouse business module of the Windows system.

[0262] In this embodiment, the Windows system's input framework can acquire keyboard and mouse events, including keyboard and mouse displacement (or position offset). The Windows system's input framework reports these events to the Windows keyboard and mouse service module. The Windows keyboard and mouse service module determines whether a keyboard and mouse movement has occurred based on the displacement.

[0263] Step 002: If the keyboard and mouse service module of the Windows system recognizes that keyboard and mouse crossing has occurred (i.e., the keyboard and mouse of the PC crosses over to the Android device) based on the received keyboard and mouse events, it sends an instruction message to the input frame of the Windows system to intercept the event.

[0264] Intercepting events are used to intercept keyboard and mouse events. The indication information of intercepting events is used to instruct the input frame to set keyboard hooks, mouse hooks, and shortcut hooks. A hook is a platform in the Windows message handling mechanism where applications can set subroutines to monitor certain messages for a specified window, and the monitored window can be created by other processes. When a message arrives, it is processed before the target window's handling function. The hook mechanism allows applications to intercept and process window messages or specific events. A hook is actually a message-processing program segment that is hooked into the system via a system call. Whenever a specific message is sent, the hook program intercepts the message before it reaches the destination window; that is, the hook function gains control first. At this point, the hook function can process (modify) the message, continue to pass the message without processing it, or forcibly terminate the message transmission.

[0265] Keyboard hooks are used to capture user input via the keyboard, such as capturing the user typing "memo information". Mouse hooks are used to capture user control actions via the mouse, such as clicking or dragging. Shortcut key hooks are used to capture user shortcut key actions, such as capturing Ctrl+C, Ctrl+V, Alt+TAB, or Ctrl+Alt+...

[0266] The DELETE operation.

[0267] By intercepting events, the mouse cursor travels from the PC to the Android device. The Windows system's input framework intercepts and listens for keyboard and mouse events, causing the business logic on the PC to not respond to these events, thus preventing the PC from affecting mouse events.

[0268] It's important to note that the Windows system's input framework has both event dispatch and data input capabilities. Data input refers to the Windows input framework reporting event-related data to application-layer business modules. Event dispatch refers to the Windows input framework reporting events to application-layer business modules. For the application layer, if a business module receives an event and its related data reported by the input framework, it can respond to the event based on the related data. For example, the related data for a keyboard and mouse event includes mouse movement. After a business module in the application layer responds to a keyboard and mouse event, it displays the mouse movement and its position on the PC screen based on the mouse movement. If a business module in the application layer receives the related data but not the event itself, it cannot respond to the event. In other words, intercepting an event is equivalent to instructing the various business modules in the Windows system's application layer not to respond to keyboard and mouse events, but it does not affect the reporting of keyboard and mouse event data (mouse movement).

[0269] Step 003: The keyboard and mouse service module of the Windows system packages the keyboard and mouse events according to the preset protocol to obtain the data packets of the keyboard and mouse events.

[0270] Specifically, the keyboard and mouse events are packaged according to a preset protocol. This involves converting the relevant data related to the keyboard and mouse events into the corresponding data format according to the data format defined in the preset protocol, and then packaging the converted data.

[0271] Optionally, the keyboard and mouse service module in Windows systems can encrypt data packets to prevent the leakage of keyboard and mouse events.

[0272] Step 004: The keyboard and mouse service module of the Windows system sends the data packets of keyboard and mouse events to the communication module of the Windows system.

[0273] Optionally, the Windows system's keyboard and mouse service module can send keyboard and mouse event data packets and keyboard and mouse traversal indication information to the Windows system's communication module. The keyboard and mouse traversal indication information is used to instruct the Android system's keyboard and mouse service module to perform keyboard and mouse traversal.

[0274] Step 005: The communication module of the Windows system sends the data packets of keyboard and mouse events to the communication module of the Android system.

[0275] Step 006: The communication module of the Android system reports the received data packets to the keyboard and mouse service module of the Android system.

[0276] Step 007: The keyboard and mouse service module of the Android system parses the data packets according to the preset protocol to obtain keyboard and mouse events.

[0277] As described in step 003, if the Windows system's keyboard and mouse service module encrypts the data packet, the Android system's keyboard and mouse service module first decrypts the encrypted data packet to obtain the data packet, and then parses the data packet according to a preset protocol to obtain the keyboard and mouse events. The data obtained by the Android system's keyboard and mouse service module includes the keyboard and mouse events and their related data (such as keyboard and mouse movement).

[0278] It should be noted that the default protocol used for parsing the keyboard and mouse service module in the Android system is the same as the default protocol used for packaging the keyboard and mouse service module in the Windows system. In some application scenarios, this default protocol can be shared after a keyboard and mouse connection is established between the PC and the Android device.

[0279] Step 008: The keyboard and mouse service module of the Android system injects keyboard and mouse events into the virtual input driver of the Android system.

[0280] Step 009: The Android system's virtual input driver reports keyboard and mouse events to the Android system's input framework.

[0281] The Android system's input framework can report keyboard and mouse events to various business modules in the Android application layer to enable keyboard and mouse functionality.

[0282] By following the steps above, keyboard and mouse can be switched between PCs and Android devices.

[0283] It should be noted that keyboard and mouse crossover between PCs and Android devices requires an existing keyboard and mouse connection between the two devices. For details on how to establish a keyboard and mouse connection, please refer to [link / reference needed]. Figure 4 The content described in the examples will not be repeated here.

[0284] To simplify the process, the following embodiments omit the data interaction process between terminal devices via the communication module, and the data interaction process between the keyboard and mouse management submodule and the communication module via the connection scheduling submodule. However, it is understood that the data interaction (such as data packets for keyboard and mouse events) between the PC's keyboard and mouse service module and the Android device's keyboard and mouse service module can be understood as the PC's keyboard and mouse service module first transmitting data to the PC's communication module, then the PC's communication module transmitting it to the Android device's communication module, and finally the Android device's communication module transmitting it to the Android device's keyboard and mouse service module. Furthermore, while the following embodiments describe the keyboard and mouse service module as a whole, it is understood that the interaction between the keyboard and mouse service module and the communication module must go through the connection scheduling submodule within the keyboard and mouse service module. That is, other submodules within the keyboard and mouse service module, excluding the connection scheduling submodule, first transmit data to the connection scheduling submodule, which then transmits it to the communication module; or, the communication module first transmits data to the connection scheduling submodule, which then transmits it to other submodules within the keyboard and mouse service module.

[0285] Based on the above implementation principle, the first method of crossing over (crossing over the keyboard and mouse of the PC to the Android device) will be introduced below.

[0286] See Figure 12 This is a schematic diagram of the interaction flow of a keyboard and mouse traversal method provided in an embodiment of this application. Figure 12 As shown, the interaction process of a PC's keyboard and mouse crossing over to an Android device can include the following steps:

[0287] S1201 establishes a keyboard and mouse connection between a PC and an Android device.

[0288] For instructions on connecting the keyboard and mouse in this step, please refer to [link / reference]. Figure 4 The descriptions in the embodiments will not be repeated here.

[0289] S1202, the PC's keyboard and mouse business module registers a listener for keyboard and mouse events with the PC's input frame.

[0290] After registering a listener for keyboard and mouse events in the PC's keyboard and mouse service module, keyboard and mouse events can be listened to.

[0291] S1203 After registering the listening event, the PC's input framework continuously reports the keyboard and mouse movement to the PC's keyboard and mouse business module.

[0292] For example, when a user presses a key on the keyboard, or moves, clicks, or drags the mouse, the PC's input frame can obtain the keyboard and mouse events and the corresponding keyboard and mouse displacement (or position offset) through the PC's input driver.

[0293] S1204, the PC's keyboard and mouse service module determines whether the current mouse pointer has reached the edge of the PC screen based on the keyboard and mouse movement.

[0294] It is understandable that S1203-S1204 can be continuous. In other words, after registering the listening event, the PC's input frame continuously reports the keyboard and mouse movement to the PC's keyboard and mouse service module, and each time the PC's keyboard and mouse service module receives the keyboard and mouse movement, it determines whether the keyboard and mouse have reached the edge of the PC's screen based on the current keyboard and mouse movement.

[0295] The process for determining the position of the keyboard and mouse can be found in the descriptions in the embodiments S1301-S1305 below.

[0296] S1205, if the keyboard and mouse are currently at the edge of the PC screen, the PC's keyboard and mouse service module obtains the position of the keyboard and mouse.

[0297] If the keyboard and mouse are not currently at the edge of the PC screen, the PC's keyboard and mouse service module will continue to listen for keyboard and mouse events.

[0298] In this step, the PC's keyboard and mouse service module can determine the crossing position based on the keyboard and mouse displacement reported by the input frame.

[0299] S1206 sends a traversal command to the keyboard and mouse service module of the Android device.

[0300] Correspondingly, the keyboard and mouse service module of the Android device receives the traversal command. The keyboard and mouse service module of the PC informs the keyboard and mouse service module of the Android device to traverse the keyboard and mouse.

[0301] S1207: After receiving the traversal command, the keyboard and mouse service module of the Android device replies with confirmation information to the keyboard and mouse service module of the PC.

[0302] S1208, the keyboard and mouse service module of the Android device determines the crossing location based on the crossing command.

[0303] In one implementation, the traversal command can include the traversal position of the keyboard and mouse on the PC screen. If the screen resolutions of the Android device and the PC are different, the mouse's position on the screen will differ when traversing between the two devices. The keyboard and mouse service module of the Android device can determine the traversal position of the mouse on the Android device screen based on the mouse's position on the PC screen.

[0304] For example, if the mouse pointer's crossing position on the PC screen is (x1, y1), and the crossing direction is left or right, the PC's conversion module can be based on s y Change the transformation of y1, i.e., y1×s y The horizontal axis value is a preset value, which is either the minimum or maximum value of the screen's side length along the X-axis. R 1y R is the side length of the Android device's screen along the y-axis. 2y This represents the side length of the PC screen along the y-axis.

[0305] If the mouse cursor's position on the PC screen is (x1, y1), and the movement direction is upward or downward, the PC's conversion module can be based on s x Change the transformation of x1, i.e., x1×s x The value of the vertical axis is a preset value, which is either the minimum or maximum value of the screen's side length on the X-axis. R 1x R is the side length of the Android device's screen along the x-axis. 2x This represents the side length of the PC screen along the x-axis.

[0306] For example, in Figure 13 In the example shown, assuming the PC's screen resolution is 1920×1080, the crossing position is (1920, 960), and the crossing direction is right, then the transformed position could be (0, 960×s). y If the crossing position is (0, 960) and the crossing direction is right, and the Android device's screen resolution is 2000×1000, then the converted position could be (2000, 960×1000). y ).in,

[0307] Understandably, in another implementation, the crossing command could include the proportion of the screen's edge length occupied by the keyboard and mouse's crossing position on the PC screen. and The keyboard and mouse service module of the Android device calculates the position of the keyboard and mouse across the screen of the Android device based on the received ratio and the side length of the Android device screen.

[0308] It should be noted that the above s x and s y This data can be calculated by the keyboard and mouse module of either the PC or the Android device after the Android device establishes a keyboard and mouse connection with the PC. This data can be interacted with via cross-commands.

[0309] S1209, the keyboard and mouse service module of the Android device updates the traversal status based on the traversal command.

[0310] The "update travel status" step refers to changing the travel status from "never traveled" to "traveling".

[0311] S1210: After receiving the confirmation information, the keyboard and mouse service module of the PC sends an instruction message for intercepting the event to the input frame of the PC.

[0312] The interception event in this step and Figure 11 Step 002 in the embodiment has the same function, see details. Figure 11 The descriptions in the embodiments will not be repeated here.

[0313] It should be noted that S1208-S1210 can be executed in any order and can be executed in parallel.

[0314] S1211, the PC's keyboard and mouse service module packages the reported keyboard and mouse events according to a preset protocol to obtain a data packet of the keyboard and mouse events.

[0315] S1212, the PC's keyboard and mouse service module sends data packets to the Android device's keyboard and mouse service module.

[0316] S1213, the keyboard and mouse service module of the Android device responds to keyboard and mouse events.

[0317] Specifically, the keyboard and mouse service module of the Android device parses the data packets to obtain the keyboard and mouse displacement of the events, updates the keyboard and mouse position based on the displacement, and injects the keyboard and mouse events into the virtual input driver of the Android device based on the updated position. The virtual input driver of the Android device reports the keyboard and mouse events to the input framework of the Android device, and the input framework of the Android device then distributes the keyboard and mouse events to various service modules of the application layer of the Android device to realize the PC keyboard and mouse function on the Android device.

[0318] Steps S1211-S1213 and Figure 11 The principle of steps 003-008 in the embodiment is the same, see details. Figure 11 The description in the embodiments, Figure 12 No further illustration is provided.

[0319] S1214, Android device keyboard and mouse service module listens for keyboard and mouse crossover.

[0320] In this embodiment, the process of the Android device's keyboard and mouse service module monitoring keyboard and mouse movement includes: the Android device's keyboard and mouse service module continuously receives data packets sent by the PC's keyboard and mouse service module, parses the keyboard and mouse events from the data packets, and injects the keyboard and mouse events into the virtual input driver; the virtual input driver reports the keyboard and mouse events to the Android device's input framework; the input framework reports the keyboard and mouse events to various service modules of the Android device's application layer, including the keyboard and mouse service module; and the keyboard and mouse service module obtains the keyboard and mouse displacement based on the reported keyboard and mouse events.

[0321] It is understandable that S1211-S1214 can be continuous during the transition from PC's keyboard and mouse to Android device. In other words, during this transition, the PC's keyboard and mouse service module continuously sends data packets to the Android device's keyboard and mouse service module, while the Android device's keyboard and mouse service module continuously listens for the transition and processes the keyboard and mouse events parsed from the data packet whenever it receives one.

[0322] Since the PC's keyboard and mouse service module continuously sends data packets to the Android device's keyboard and mouse service module, this can be more figuratively understood as the PC's keyboard and mouse service module sending data streams to the Android device's keyboard and mouse service module.

[0323] After the keyboard and mouse have been transferred to an Android device, they can also be transferred back. In the scenario of transferring the keyboard and mouse back to the PC, the keyboard and mouse transfer operation can be understood as the keyboard and mouse transfer back operation, that is, transferring from the Android device back to the PC. The following is a further explanation.

[0324] S1215, the keyboard and mouse service module of the Android device determines whether the keyboard and mouse have reached the edge of the Android device screen based on the monitored keyboard and mouse displacement.

[0325] The process for determining the position of the keyboard and mouse can be found in the descriptions in the embodiments S1301-S1305 below.

[0326] S1216 If the current keyboard and mouse reach the edge of the Android device's screen, send a message to the PC's keyboard and mouse service module to end the crossing.

[0327] S1217, the PC's keyboard and mouse service module receives the information to end the traversal and sends a stop instruction to the PC's input frame to stop the interception event.

[0328] Upon receiving the stop instruction for the interception event, the PC's input framework ceases intercepting keyboard and mouse events. After stopping the interception, the PC's input framework redistributes keyboard and mouse events to the various business modules in the PC's application layer.

[0329] S1218, PC's keyboard and mouse service module has been restored.

[0330] In this step, restoring the status includes updating the traversal status and restoring the keyboard and mouse positions. Updating the traversal status refers to changing the traversal status from "traversing" to "not traversing".

[0331] One way to restore the keyboard and mouse positions is to restore the PC's keyboard and mouse positions based on the positions of the keyboard and mouse when they return from the Android device to the PC screen.

[0332] Another way to restore the keyboard and mouse positions is to restore the PC's keyboard and mouse positions based on the positions the keyboard and mouse have moved from the PC to the Android device screen.

[0333] Another way to restore the keyboard and mouse positions is to restore them to a preset location. For example, the preset location is the center of the PC screen.

[0334] S1219, the PC's keyboard and mouse service module sends a setup completion message to the Android device's keyboard and mouse service module.

[0335] Accordingly, the keyboard and mouse service module of the Android device receives the setting completion message. This setting completion message indicates that the keyboard and mouse traversal has ended.

[0336] It should be noted that in this embodiment, the order of S1219 and S1217 is not limited, and they can be executed in parallel.

[0337] S1220, the keyboard and mouse service module of the Android device updates the traversal status based on the information that the settings have been completed.

[0338] In this step, updating the crossing status means changing the crossing status from "crossed" to "not crossed".

[0339] The keyboard and mouse crossing method provided in this application embodiment allows the keyboard and mouse crossing positions to be adjusted based on different devices after the keyboard and mouse cross-device crossing is achieved. After the mouse cross-device crossing is successful, the keyboard and mouse events are intercepted on the PC and responded on the Android device side, which can realize the cross-device crossing of keyboard and mouse.

[0340] The above method involves determining the keyboard and mouse positions in steps S1204 and S1215. A method for detecting the mouse position is described below. The mouse position detection method may include the following steps:

[0341] S1301, Keyboard and mouse business module obtains the current position of the mouse.

[0342] The current position of the mouse is its coordinates on the screen. The keyboard and mouse module can create a two-dimensional coordinate system on the screen, and the mouse's coordinates in the two-dimensional coordinate system are the current position of the mouse.

[0343] For example, Figure 13 A schematic diagram illustrating the establishment of a screen coordinate system is shown. (For example...) Figure 13 As shown, a coordinate system is established on screen OABC with point O as the origin, the direction of OA as the positive X-axis, and the direction of OC as the positive Y-axis. The screen resolution is 1920×1080. The length of line segment OA is equal to the length of line segment CB (corresponding to 1920 pixels), and the length of line segment OC is equal to the length of line segment AB (corresponding to 1080 pixels). Therefore, in the coordinate system, the coordinates of point O are (0,0), the coordinates of point A are (1920,0), the coordinates of point B are (1920,1080), and the coordinates of point C are (0,1080). The mouse position is the mouse's coordinate in the coordinate system; for example, the mouse position could be (1920,960).

[0344] S1302, Keyboard and mouse business module obtains mouse displacement.

[0345] For example, when a user moves or drags the mouse, the PC's virtual input driver reports keyboard and mouse events to the PC's input frame. The PC's input frame obtains the mouse displacement based on the keyboard and mouse events and reports the mouse displacement to the PC's keyboard and mouse business module.

[0346] Mouse displacement refers to the distance the mouse moves. Mouse displacement can be represented by (Lx, Ly). Lx and Ly can be positive or negative values, and this embodiment does not limit this.

[0347] S1303, the keyboard and mouse business module calculates the position of the mouse after it has moved based on the mouse position and mouse displacement.

[0348] For example, the current mouse position can be represented by (XRaw, YRaw), and the position after the mouse has moved can be represented by (curX, curY). The position after the mouse has moved can be expressed by the following formulas: curX = XRaw + Lx; curY = YRaw + Ly.

[0349] S1304, The keyboard and mouse business module determines whether the position of the mouse after movement exceeds the screen.

[0350] In scenarios where the mouse moves across the screen horizontally, if curX is less than the minimum value of the screen on the X-axis or greater than the maximum value of the screen on the X-axis, the PC's keyboard virtualization module can determine that the mouse is outside the screen.

[0351] In scenarios where the mouse moves vertically across the screen, if curY is less than the minimum value of the screen on the Y-axis or greater than the maximum value of the screen on the Y-axis, the PC's keyboard virtualization module can determine that the mouse is outside the screen.

[0352] If the mouse moves beyond the screen, it can be determined that the mouse has crossed the screen, and S1305 is executed. If the mouse moves within the screen, the current position is determined and the mouse movement is continued, and S1302 is executed.

[0353] S1305. If the position of the mouse after movement exceeds the screen, the keyboard and mouse service module can determine that the mouse has crossed the screen.

[0354] The keyboard and mouse business module can also determine the direction of mouse movement. For example, in a scenario where the mouse moves left or right across the screen, the movement direction can be left or right. In a scenario where the mouse moves up or down across the screen, the movement direction can be up or down.

[0355] For example, in the above Figure 13 In the example shown, when the mouse moves left and right across the screen, if curX is less than 0, the keyboard and mouse application module can determine that the mouse is moving left. If curX is greater than 1920, the module can determine that the mouse is moving right. When the mouse moves up and down across the screen, if curY is less than 0, the module can determine that the mouse is moving up. If curX is greater than 1080, the module can determine that the mouse is moving down.

[0356] It should be noted that the methods described in S1301-S1305 can be executed by a PC or an Android device. For example, in S1204, the keyboard and mouse service module of the PC executes the methods in S1301-S1305. In S1215, the keyboard and mouse service module of the Android device executes the methods in S1301-S1305.

[0357] The mouse crossing detection method provided in this application can determine whether the mouse has exceeded the screen and the crossing direction by measuring the mouse position and displacement, which helps to improve the accuracy of detecting mouse crossing.

[0358] As an example of the second method of data transfer (porting keyboard and mouse from an Android device to a PC), see [link to relevant documentation]. Figure 14 This is a schematic diagram of another keyboard and mouse traversal scenario provided in an embodiment of this application. Figure 14 The keyboard and mouse traversal scenario shown can be applied to Figure 1 In the communication system shown, such as Figure 1 As shown, the right side of computer 11 is connected to mobile phone 12, and the left side is connected to tablet 13.

[0359] like Figure 14 As shown in (a), when the user manipulates the mouse on the mobile phone 12 to move the mouse pointer 121 on the screen of the mobile phone 12 to the left, as... Figure 14As shown in (a) and (b), keyboard and mouse movement is possible between the mobile phone 12 and the computer 11. In other words, the mouse pointer 121 displayed on the mobile phone 12 can move from the screen of the mobile phone 12 to the left across the screen of the computer 11. When the mouse pointer 121 of the mobile phone 12 moves across the screen of the computer 11, as shown in (a) and (b), the mouse cursor can move across the screen of the computer 11. Figure 14 As shown in (b) above, the mouse pointer 121 can be displayed as an arrow on the screen of computer 11. Figure 14 As shown in (b), the user can also move the mouse pointer 121 on the computer screen to the right by operating the mouse on the mobile phone 12, such as... Figure 14 As shown in (b) and (c), the mouse pointer 121 on the computer screen 11 moves to the right and crosses back onto the screen of the mobile phone 12, as shown in (b) and (c). Figure 14 As shown in (c), the mouse pointer 121 is displayed as a circle on the screen of the mobile phone 12.

[0360] In this example, mouse pointer 121 is displayed as a circle on phone 12. Mouse pointer 121 travels from phone 12 to computer 11, where it is displayed as an arrow. Mouse pointer 121 travels back from computer 11 to phone 12, where it is again displayed as a circle. When an external mouse is connected to computer 11, computer 11 also displays the mouse pointer as an arrow. In other examples, the mouse pointer may display in other forms after traveling from phone to computer; this is not limited here.

[0361] After the keyboard and mouse from phone 12 are transferred to computer 11, the user can use the mouse to click, drag, or double-click on computer 11, and use the keyboard to input information on computer 11. No further illustration is provided here.

[0362] To better understand the embodiments of this application, the following is combined with... Figure 2 and Figure 3 The software architecture shown illustrates the implementation principle of the second traversal method (traversing the keyboard and mouse of an Android device to a PC).

[0363] See Figure 15 This is a schematic diagram illustrating the implementation principle of a traversal method provided in an embodiment of this application. For example... Figure 15 As shown, the implementation principle of enabling keyboard and mouse functionality on an Android device to migrate to a PC can include the following steps:

[0364] Step 010: The Windows system's keyboard and mouse service module loads the virtual driver.

[0365] like Figure 2 and Figure 15As shown, the driver layer of the Windows system includes a virtual driver and an input driver. The virtual driver is used to drive the keyboard and mouse on Android devices, while the input driver is used to drive the keyboard and mouse on PCs. In this step, the keyboard and mouse service module of the Windows system loads the virtual driver to provide driver support for the subsequent crossover of keyboard and mouse functionality from Android devices to PCs.

[0366] like Figure 2 As shown, the keyboard and mouse service module of the Windows system includes a virtual driver loading submodule. Accordingly, step 010 can be performed by loading the virtual driver by the virtual driver loading submodule of the Windows system.

[0367] It should be noted that step 010 can be performed after the PC and Android device establish a keyboard and mouse connection, or it can be performed before step 019. There is no limitation on this.

[0368] Step 011: The virtual input driver of the Android system detects the keyboard and mouse events of the Android device and reports the keyboard and mouse events of the Android system to the Android input framework.

[0369] Step 012: The input framework of the Android system reports the received keyboard and mouse events to the keyboard and mouse business module of the Android system.

[0370] In this embodiment, the Android system's input framework can obtain the keyboard and mouse displacement (or position offset) corresponding to the keyboard and mouse events. The Android system's input framework reports the keyboard and mouse events and their corresponding displacements to the Android keyboard and mouse service module. The Android keyboard and mouse service module determines whether a keyboard and mouse crossing has occurred based on the displacement.

[0371] Step 013: If the keyboard and mouse service module of the Android system recognizes that keyboard and mouse crossing has occurred (i.e., the keyboard and mouse of the Android device crosses over to the PC screen) based on the received keyboard and mouse events, it calls the preset interface of the input framework of the Android system to intercept the reporting of keyboard and mouse events.

[0372] Unlike Windows systems, Android intercepts keyboard and mouse events by calling a pre-defined interface of the input framework, rather than by intercepting the events themselves. By calling this pre-defined interface, the mouse cursor travels from the Android device to the PC. The Android system's input framework intercepts and listens for these keyboard and mouse events, preventing the applications on the Android device from responding to them and thus ensuring that the Android device does not interfere with the keyboard and mouse events.

[0373] The principle / function of calling the input frame's preset interface in the Android system is the same as the principle / function of the input frame's event interception in the Windows system. For details, please refer to [link / reference needed]. Figure 11 The description of step 002 in the embodiment will not be repeated here.

[0374] Step 014: The keyboard and mouse service module of the Android system packages the keyboard and mouse events according to the preset protocol to obtain the data packets of the keyboard and mouse events.

[0375] Optionally, the keyboard and mouse service module of the Android system can encrypt data packets to prevent the leakage of keyboard and mouse events.

[0376] Step 015: The keyboard and mouse service module of the Android system sends the data packets of keyboard and mouse events to the communication module of the Android system.

[0377] Optionally, the Android system's keyboard and mouse service module can send keyboard and mouse event data packets and keyboard and mouse traversal indication information to the Android system's communication module. The keyboard and mouse traversal indication information is used to instruct the Windows system's keyboard and mouse service module to perform keyboard and mouse traversal.

[0378] Step 016: The Android system's communication module sends the data packets of keyboard and mouse events to the Windows system's communication module.

[0379] Step 017: The communication module of the Windows system reports the received data packets to the keyboard and mouse service module of the Windows system.

[0380] Step 018: The keyboard and mouse service module of the Windows system parses the data packets according to the preset protocol to obtain the keyboard and mouse events.

[0381] As described in step 014, if the Android system's keyboard and mouse service module encrypts the data packet, the Windows system's keyboard and mouse service module first decrypts the encrypted data packet to obtain the data packet, and then parses the data packet according to a preset protocol to obtain the keyboard and mouse events. The data obtained by the Windows system's keyboard and mouse service module includes the keyboard and mouse events and their related data (such as keyboard and mouse movement).

[0382] It should be noted that the default protocol used for parsing the keyboard and mouse application modules in Windows is the same as the default protocol used for packaging the keyboard and mouse application modules in Android. In some application scenarios, this default protocol can be shared after a keyboard and mouse connection is established between a PC and an Android device.

[0383] Step 019: The Windows system's keyboard and mouse service module injects keyboard and mouse events into the Windows system's virtual driver.

[0384] Step 020: The virtual driver of the Windows system reports keyboard and mouse events to the Windows system's input frame.

[0385] The Windows system's input framework can report keyboard and mouse events to various business modules in the Windows application layer to enable keyboard and mouse functionality.

[0386] By following the steps above, keyboard and mouse can be switched between Android devices and PCs.

[0387] It should be noted that keyboard and mouse crossover between Android devices and PCs requires an existing keyboard and mouse connection to be established between the two devices. For details on establishing a keyboard and mouse connection, please see [link to documentation / section]. Figure 4 The content described in the examples will not be repeated here.

[0388] Based on the above implementation principle, the second method of traversal (traversing the keyboard and mouse from an Android device to a PC) is introduced below. See [link / reference] Figure 16 This is a schematic diagram of the interaction flow of a keyboard and mouse traversal method provided in an embodiment of this application. Figure 16 As shown, the interaction process of keyboard and mouse on an Android device crossing over to a PC can include the following steps:

[0389] S1601 establishes a keyboard and mouse connection between a PC and an Android device.

[0390] For instructions on connecting the keyboard and mouse in this step, please refer to [link / reference]. Figure 4 The descriptions in the embodiments will not be repeated here.

[0391] S1602, The keyboard and mouse service module of the Android device registers a listener for keyboard and mouse events with the input framework of the Android device.

[0392] After registering a listener for keyboard and mouse events, the keyboard and mouse service module on an Android device can listen for these events.

[0393] S1603 After registering the listening event, the input framework of the Android device continuously reports the keyboard and mouse movement to the keyboard and mouse business module of the Android device.

[0394] For example, when a user presses a key on the keyboard, or moves, clicks, or drags the mouse, the Android device's input framework can obtain the keyboard and mouse events and the corresponding keyboard and mouse displacements (or position offsets) through the Android device's virtual input driver.

[0395] S1604, the keyboard and mouse service module of the Android device determines whether the current mouse pointer has reached the edge of the Android device screen based on the keyboard and mouse displacement.

[0396] It is understandable that S1603-S1604 can be continuous. In other words, after registering the listening event, the input framework of the Android device continuously reports the keyboard and mouse movement to the keyboard and mouse service module of the Android device. Each time the keyboard and mouse service module of the Android device receives the keyboard and mouse movement, it determines whether the keyboard and mouse have reached the edge of the Android device screen based on the current keyboard and mouse movement.

[0397] The process for determining the position of the keyboard and mouse can be found in the description of the method embodiments in S1301-S1305 above, and will not be repeated here.

[0398] S1605: If the keyboard and mouse are currently at the edge of the Android device's screen, obtain the position where the keyboard and mouse have crossed.

[0399] If the keyboard and mouse are not currently at the edge of the Android device's screen, the Android device's keyboard and mouse service module will continue to listen for keyboard and mouse events.

[0400] S1606 sends a traversal command to the PC's keyboard and mouse service module.

[0401] Correspondingly, the PC's keyboard and mouse service module receives the traversal command. The Android device's keyboard and mouse service module informs the PC's keyboard and mouse service module to traverse the keyboard and mouse.

[0402] S1607: After receiving the traversal command, if the current traversal status is not traversal, the PC's keyboard and mouse service module replies with confirmation information to the Android device's keyboard and mouse service module.

[0403] S1608, the PC's keyboard and mouse business module determines the crossing position based on the crossing command.

[0404] In this step, the PC's keyboard and mouse service module determines the traversal location using the same method as... Figure 12 The principle of step S1208 in the embodiment is the same, and the details can be found in the description in S1208, which will not be repeated here.

[0405] S1609, the PC's keyboard and mouse service module updates the traversal status based on the traversal command.

[0406] The "update travel status" step refers to changing the travel status from "never traveled" to "traveling".

[0407] S1610 After receiving the confirmation information, the keyboard and mouse service module of the Android device calls the preset interface of the input framework of the Android device to intercept the reporting of keyboard and mouse events.

[0408] The preset interface in this step and Figure 15 Step 013 in the embodiment has the same function, see details. Figure 15 The descriptions in the embodiments will not be repeated here.

[0409] It should be noted that S1608-S1610 can be executed in any order and can be executed in parallel.

[0410] S1611, the keyboard and mouse service module of the Android device packages the reported keyboard and mouse movement according to the preset protocol to obtain the data packet of the keyboard and mouse event.

[0411] S1612, the keyboard and mouse service module of the Android device sends data packets to the keyboard and mouse service module of the PC.

[0412] S1613, the PC's keyboard and mouse service module responds to keyboard and mouse events.

[0413] Specifically, the PC's keyboard and mouse service module parses the data packets to obtain the keyboard and mouse movement of the events, updates the keyboard and mouse position based on the movement, and injects the events into the PC's virtual driver based on the updated position. The PC's virtual driver reports the events to the PC's input framework, which then distributes the events to various service modules in the PC's application layer to enable the keyboard and mouse functionality of Android devices on the PC.

[0414] Steps S1611-S1613 and Figure 15 The principle of steps 014-020 in the embodiment is the same, see details. Figure 15 The descriptions in the embodiments will not be repeated here.

[0415] S1614, PC's keyboard and mouse service module monitors keyboard and mouse movement.

[0416] In this embodiment, the process of the PC's keyboard and mouse service module monitoring keyboard and mouse movement includes: the PC's keyboard and mouse service module continuously receives data packets sent by the Android device's keyboard and mouse service module, parses the data packets to determine the keyboard and mouse displacement corresponding to the keyboard and mouse events, updates the keyboard and mouse positions based on the displacement, and injects the updated positions into the virtual driver to update the keyboard and mouse positions on the PC screen. When the PC's keyboard and mouse service module detects keyboard and mouse movement, it can obtain the keyboard and mouse displacement corresponding to the movement.

[0417] Understandably, S1611-S1614 can be continuous during the transition from Android device's keyboard and mouse functionality to PC. In other words, during this transition, the Android device's keyboard and mouse service module continuously sends data packets to the PC's keyboard and mouse service module, while the PC's module continuously listens for the transition and processes the keyboard and mouse events parsed from the data packet whenever it receives one.

[0418] Since the keyboard and mouse service module of the Android device continuously sends data packets to the keyboard and mouse service module of the PC, this can also be more figuratively understood as the keyboard and mouse service module of the Android device sending data streams to the keyboard and mouse service module of the PC.

[0419] After the keyboard and mouse have moved to the PC, they can also move back. In the scenario of moving the keyboard and mouse back to the Android device, the keyboard and mouse movement can be understood as the keyboard and mouse returning, that is, moving back from the PC to the Android device. The following will explain further.

[0420] S1615, the PC's keyboard and mouse service module determines whether the keyboard and mouse have reached the edge of the PC screen based on the monitored keyboard and mouse movement.

[0421] The process for determining the position of the keyboard and mouse can be found in the description of the method embodiments in S1301-S1305 above, and will not be repeated here.

[0422] S1616 If the current keyboard and mouse reach the edge of the PC screen, send a message to the keyboard and mouse service module of the Android device to end the crossing.

[0423] S1617, the keyboard and mouse service module of the Android device receives the message to end the traversal and stops calling the preset interface of the input framework of the Android device to stop intercepting the reporting of keyboard and mouse events.

[0424] After stopping calls to the preset API, the Android device's input framework stops intercepting keyboard and mouse events. After stopping event interception, the Android device's input framework redistributes keyboard and mouse events to various business modules in the application layer of the Android device.

[0425] S1618, the keyboard and mouse service module of the Android device has been restored.

[0426] In this step, restoring the status includes updating the traversal status and restoring the keyboard and mouse positions. Updating the traversal status refers to changing the traversal status from "traversed" to "not traversed".

[0427] One way to restore the keyboard and mouse position is to restore the keyboard and mouse position on the Android device based on the position of the keyboard and mouse when they move from the PC back to the screen of the Android device.

[0428] Another way to restore the keyboard and mouse position is to restore the keyboard and mouse position on the Android device based on the position of the keyboard and mouse as they move from the Android device to the PC screen.

[0429] Another way to restore the keyboard and mouse positions is to restore the Android device's keyboard and mouse positions based on preset locations. For example, the mouse position can be restored to the center of the Android device screen.

[0430] S1619, the keyboard and mouse service module of the Android device sends a setup completion message to the keyboard and mouse service module of the PC.

[0431] Accordingly, the PC's keyboard and mouse service module receives a message indicating that the setup is complete. This message signifies that the keyboard and mouse traversal has ended.

[0432] It should be noted that in this embodiment, the order of S1619 and S1617 is not limited, and they can be executed in parallel.

[0433] S1620, the PC's keyboard and mouse service module updates the traversal status based on the setup completion information.

[0434] In this step, updating the crossing status means changing the crossing status from "crossed" to "not crossed".

[0435] The keyboard and mouse crossing method provided in this application embodiment allows the keyboard and mouse crossing positions to be adjusted based on different devices after the keyboard and mouse cross-device crossing is achieved. After the mouse cross-device crossing is successful, the keyboard and mouse events are intercepted on the Android device and responded on the PC side, thus enabling keyboard and mouse to cross-device crossing.

[0436] like Figure 12 Examples and Figure 16 As shown in the examples, both share the premise that a keyboard and mouse connection has been established between the PC and the Android device, and this connection is centered on the PC. The difference lies in... Figure 12 In this context, the PC is the initiator of the keyboard and mouse crossover event, while the Android device is the receiver. Figure 16 In this scenario, the Android device is the initiator of the keyboard and mouse crossover event, while the PC is the receiver.

[0437] Based on the above Figures 12 to 16 The two keyboard and mouse switching methods described in the embodiments allow users to flexibly operate multiple terminal devices with the same set of keyboard and mouse, avoiding the need to switch between operating methods between multiple terminal devices, simplifying the operation and improving the user experience.

[0438] The above Figure 9 and Figure 14 The application scenario shown illustrates keyboard and mouse movement between two terminal devices. In other application scenarios, users may need to operate three or more terminal devices, which involves keyboard and mouse movement between multiple terminal devices.

[0439] As an example of keyboard and mouse traversal between multiple terminal devices, this example illustrates traversal between three terminal devices. See [link / reference] Figure 17 and Figure 18 This is a schematic diagram of a keyboard and mouse traversing a scene provided in an embodiment of this application.

[0440] Figure 17 and Figure 18 The keyboard and mouse traversal scenario shown can be applied to Figure 1 In the communication system shown, such as Figure 1 As shown, the right side of computer 11 is connected to mobile phone 12, and the left side is connected to tablet 13.

[0441] like Figure 17 As shown in (a), when the user moves the mouse pointer on the screen of mobile phone 12 to the left, as... Figure 17 As shown in (a) and (b), keyboard and mouse movement is possible between the mobile phone 12 and the computer 11; in other words, the mouse pointer 121 displayed on the mobile phone 12 can move from the screen of the mobile phone 12 to the left across the screen of the computer 11. Figure 17 As shown in (b), when the user continues to manipulate the mouse on the mobile phone 12 to move the mouse pointer on the computer 11 screen to the left, as... Figure 17 As shown in (b) and (c), keyboard and mouse can cross between computer 11 and tablet 13. In other words, the mouse pointer 121 displayed on computer 11 can continue to cross from the screen of computer 11 to the left to the screen of tablet 13.

[0442] continue Figure 17 In the scenario shown in (c), the current mouse pointer 121 is on the screen of tablet 13. Figure 18 As shown in (a), when the user moves the mouse pointer on the tablet 13 screen to the right, as... Figure 18 As shown in (a) and (b), keyboard and mouse movement is possible between tablet 13 and computer 11; in other words, the mouse pointer 121 displayed on tablet 13 can move from the screen of tablet 13 to the right across the screen of computer 11. Figure 18 As shown in (b), when the user continues to move the mouse pointer on the computer 11 screen to the right, keyboard and mouse movement can be achieved between the computer 11 and the mobile phone 12. In other words, the mouse pointer 121 displayed on the computer 11 can move from the screen of the computer 11 back to the screen of the mobile phone 12, as shown in (b). Figure 18 As shown in (c) in the figure.

[0443] exist Figure 17 and Figure 18 In the application scenario shown, the mouse pointer of mobile phone 12 travels multiple times between computer 11, mobile phone 12 and tablet 13, realizing cross-PC keyboard and mouse crossing.

[0444] In application scenarios where multiple Android devices cross-PC via keyboard and mouse events, centered around a PC, the keyboard and mouse events detected by Android device I might be its own events (in which case Android device I is not in a cross-PC state); or they might be events from Android device II or the PC itself (in which case Android device I is in a cross-PC state). Failure to distinguish between these could lead to confusion regarding the cross-PC state.

[0445] Based on this consideration, this application provides a keyboard and mouse traversal method that can distinguish between keyboard and mouse traversal scenarios of multiple terminal devices, avoid the occurrence of traversal state confusion, and thus ensure the reliability of keyboard and mouse traversal.

[0446] like Figure 17 and Figure 18 In the application scenario shown, multiple keyboard and mouse pointer transfers occur between the computer 11, mobile phone 12, and tablet 13. This can be divided into two scenarios: the first scenario involves the mouse pointer transferring from the PC screen to the Android device screen; the second scenario involves the mouse pointer transferring from the Android device screen to the PC screen. The methods for keyboard and mouse pointer transfer in these two scenarios are described below.

[0447] It should be noted that, as Figure 17 and Figure 18 In the application scenario shown, keyboard and mouse connections may cross between the PC and both Android devices. Therefore, the following embodiments assume that a keyboard and mouse connection has been established between the PC and each of the two Android devices. The method for establishing a keyboard and mouse connection can be found above. Figure 4 The descriptions in the embodiments will not be repeated here.

[0448] The following describes the keyboard and mouse traversal method for the first scenario. See [link / reference] Figure 19 This is a schematic diagram of the interaction flow of a keyboard and mouse traversal method provided in an embodiment of this application. Figure 19 The method described can be applied to situations where the mouse pointer moves from a PC screen to an Android device screen. This is intended as an example, not a limitation. Figure 19 As shown, the keyboard and mouse traversal method may include the following steps:

[0449] S1901, the keyboard and mouse service module of Android device I listens for keyboard and mouse events and the movement of the keyboard and mouse.

[0450] This step is the same as S1214 in principle, and you can refer to the description of S1214 for details, which will not be repeated here.

[0451] S1902, the keyboard and mouse service module of Android device I determines whether the current mouse pointer has reached the edge of the screen based on the monitored keyboard and mouse movement.

[0452] The method for determining whether the mouse pointer has reached the edge of the screen can be found in the descriptions of the embodiments S1301-S1305 above, and will not be repeated here.

[0453] S1903, if the current mouse pointer reaches the edge of the screen, determine whether there is a device that can be passed through the current screen edge.

[0454] For example, in Android device I is Figure 17In the case of tablet 13 shown, since Android device I is on the left side of the PC, there are no devices that can cross to the left of Android device I. Therefore, if the current mouse pointer reaches the left edge of the screen, there is no need to consider keyboard and mouse crossing; if the current mouse pointer reaches the right edge of the screen, S1904 is executed. When Android device I is... Figure 17 In the case of the mobile phone 12 shown, since Android device I is on the right side of the PC, there is no device that can cross over to the right side of Android device I. Therefore, if the current mouse pointer reaches the right edge of the screen, there is no need to consider keyboard and mouse crossing; if the current mouse pointer reaches the left edge of the screen, S1904 is executed.

[0455] S1904, if there is a device that can pass through the current screen edge, determine whether the current mouse pointer belongs to the local keyboard and mouse (Android device I's keyboard and mouse).

[0456] Depend on Figure 17 and Figure 18 As shown in the application scenario, the mouse pointer on the screen of an Android device may be the mouse pointer of its own keyboard and mouse, or it may be a mouse pointer that has been transferred from a PC.

[0457] If the current mouse pointer belongs to the local keyboard and mouse (Android device I keyboard and mouse), then it is Figure 14 The image shows how to transfer the keyboard and mouse from an Android device to a PC. For instructions on how to transfer the keyboard and mouse, please refer to [link / reference needed]. Figure 16 Steps S1605-S1615 in the embodiment will not be repeated here.

[0458] It should be noted that, Figure 14 The application scenario shown only involves keyboard and mouse movement between a PC and an Android device, and... Figure 14 The application scenarios shown are different. Figure 17 and Figure 18 The application scenario shown involves keyboard and mouse traversal between a PC and multiple Android devices. Therefore, before the keyboard and mouse on Android device I traverse to the PC, the keyboard and mouse on the PC may have already traversed to Android device II. In this case, the PC is in a traversal state, meaning that the PC's input frame is executing event interception to block the reporting of keyboard and mouse events.

[0459] To ensure that the PC can execute keyboard and mouse events after the keyboard and mouse of Android device I are transferred to the PC, in one embodiment, after S1606, the keyboard and mouse service module of the PC receives the transfer instruction sent by Android device I and sends a stop instruction to the input frame of the PC to stop intercepting events, so as to instruct the input frame of the PC to stop intercepting keyboard and mouse events.

[0460] If the current mouse pointer does not belong to the local keyboard and mouse, it means that the current mouse pointer is crossing from the PC screen, then proceed to step S1905.

[0461] In some embodiments, during the keyboard and mouse traversal process, the device ID of the device to which the keyboard and mouse belong can be packaged into a data packet along with the keyboard and mouse events. For example, when the keyboard and mouse from Android device I traverse to a PC, the keyboard and mouse service module of Android device I packages the device ID of Android device I along with the keyboard and mouse events into a data packet and sends it to the keyboard and mouse service module of the PC. The keyboard and mouse service module of the PC parses the keyboard and mouse events and the device ID from the received data packet, and can confirm that the current keyboard and mouse originated from Android device I based on the device ID.

[0462] Accordingly, one way to determine whether a keyboard and mouse are from the local machine can be:

[0463] The keyboard and mouse service module obtains the device ID corresponding to the current keyboard and mouse event; and determines whether it is a local keyboard and mouse based on the device ID.

[0464] It should be noted that when an Android device determines whether it uses a local keyboard and mouse, the above determination steps are performed by the Android device's keyboard and mouse service module. When a PC device determines whether it uses a local keyboard and mouse, the above determination steps are performed by the PC's keyboard and mouse service module.

[0465] In other implementations, during the keyboard and mouse traversal process, the device ID can be carried in the data packet of each transmitted keyboard and mouse event; alternatively, the device ID can be carried in the data packet of the first transmitted keyboard and mouse event, but not in the data packets of subsequent transmitted keyboard and mouse events. In still other implementations, the device ID can be carried in the traversal command sent when the first keyboard and mouse traversal occurs.

[0466] In other embodiments, the direction of keyboard and mouse movement can be packaged into a data packet along with the keyboard and mouse events. For example, when the keyboard and mouse from Android device I move to the PC, with Android device I on the left side of the PC, the movement direction is to the right. The keyboard and mouse service module of Android device I packages the movement direction (to the right) and the keyboard and mouse events into a data packet and sends it to the keyboard and mouse service module of the PC. The keyboard and mouse service module of the PC parses the keyboard and mouse events and the movement direction (to the right) from the received data packet, and can confirm that the keyboard and mouse have moved from Android device I based on this movement direction.

[0467] like Figure 17 and Figure 18In the illustrated application scenario, the keyboard and mouse on Android device II first travel to the PC, then to Android device I. When the mouse pointer travels back from Android device I to the PC, judging solely by the direction of travel might misjudge that the keyboard and mouse belong to Android device I. To prevent misjudgment, in some embodiments, the direction of travel and the device ID of the keyboard and mouse can be packaged together with the keyboard and mouse event into a data packet. This method allows for determining the terminal to which the keyboard and mouse belong, both by the direction of travel and by the device ID. This dual judgment effectively prevents misjudgment and facilitates clearer differentiation of different travel scenarios.

[0468] In other implementations, a preset function can be called to determine whether the keyboard and mouse are the user's own or virtual keyboard and mouse. For example, the preset function can return a function value; when the function value is false, it indicates that the keyboard and mouse are virtual; when the function value is true, it indicates that the keyboard and mouse are the user's own.

[0469] S1905, if the current mouse pointer does not belong to the local keyboard and mouse (Android device I's keyboard and mouse), the keyboard and mouse service module of Android device I sends a return instruction (second instruction) to the keyboard and mouse service module of the PC and ends the traversal.

[0470] The keyboard and mouse service module of Android device I can complete the traversal according to steps S1617-S1619.

[0471] Correspondingly, the PC's keyboard and mouse service module receives this return instruction. The Android device I's keyboard and mouse service module uses this return instruction to inform the PC's keyboard and mouse service module that the mouse pointer has returned to the PC.

[0472] S1906, the PC's keyboard and mouse service module receives the return instruction and determines whether the currently traversed mouse pointer belongs to the local keyboard and mouse (PC's keyboard and mouse).

[0473] The principle for determining whether the keyboard and mouse are from the local machine in this step is the same as in S1904. For details, please refer to the description in S1904, which will not be repeated here.

[0474] If the currently traversing mouse pointer belongs to the local keyboard and mouse (PC keyboard and mouse), then it is Figure 9 The example shown illustrates a scenario where the keyboard and mouse on a PC are transferred from an Android device back to the PC. For instructions on how to transfer the keyboard and mouse, please refer to [link to documentation / reference]. Figure 16 Steps S1616-S1620 in the embodiment will not be described again here.

[0475] If the currently traversed mouse pointer does not belong to the local keyboard and mouse (PC keyboard and mouse), it means that the current mouse pointer is traversed from Android device II, then S1907 is executed.

[0476] It should be noted that, in this embodiment, when the PC's keyboard and mouse service module determines that the current mouse pointer does not belong to the local keyboard and mouse, the PC's keyboard and mouse service module can determine whether the currently traversed keyboard and mouse belongs to Android device I or Android device II based on the type of instruction received from Android device I. For example, if a traversal instruction is received, it means that the currently traversed keyboard and mouse belongs to Android device I; if a return instruction is received, it means that the currently traversed keyboard and mouse belongs to Android device II. Of course, the device ID in the data packet can also be used for collaborative determination.

[0477] S1907, the PC's keyboard and mouse service module sends a switching notification (third instruction) to the Android device II's keyboard and mouse service module.

[0478] S1908, the keyboard and mouse service module of Android device II receives a switching notification and sends the data packet of keyboard and mouse events to the keyboard and mouse service module of PC.

[0479] Afterwards, the PC's keyboard and mouse service module listens for the data packets of keyboard and mouse events and executes steps S1613-S1615.

[0480] The situation corresponding to this step is... Figure 14 The same applies to the transfer of keyboard and mouse from the Android device to the PC. This step is based on the same principle as S1613-S1615. For details, please refer to the descriptions of S1613-S1615, which will not be repeated here.

[0481] It should be noted that, in order to simplify the steps, Figure 19 The diagram only shows a portion of the software architecture in PCs and Android devices. This does not mean that other modules not shown do not participate in keyboard and mouse traversal. Modules participating in keyboard and mouse traversal can be found in the descriptions of other embodiments.

[0482] For example, in some application scenarios, Figure 19 In this embodiment, the PC can be a second device, Android device I can be a first device, and Android device II can be a third device; or, the PC can be a second device, Android device II can be a first device, and Android device I can be a third device. In this application scenario, the second device is communicatively connected to both the first device and the third device.

[0483] In this embodiment, in application scenarios where keyboard and mouse crossing occurs between multiple terminal devices, when the mouse pointer crosses from Android device I to the PC screen, the system on Android device I determines whether it is the local keyboard and mouse, thus distinguishing whether the keyboard and mouse from Android device I are crossing to the PC or from Android device I back to the PC. Similarly, when the keyboard and mouse cross from Android device I back to the PC, the system on the PC determines whether it is the local keyboard and mouse, thus distinguishing whether the keyboard and mouse are crossing from Android device I back to the PC or from Android device II to the PC. This method distinguishes scenarios involving keyboard and mouse crossing between multiple terminal devices, preventing confusion in the crossing status and ensuring the reliability of keyboard and mouse crossing.

[0484] The second method for bypassing keyboard and mouse input is described below. See [link / reference] Figure 20 This is a schematic diagram of the interaction flow of a keyboard and mouse traversal method provided in an embodiment of this application. Figure 20 The method described can be applied to situations where the mouse pointer crosses from the screen of an Android device to the screen of a PC. This is intended as an example, not a limitation. Figure 20 As shown, the keyboard and mouse traversal method may include the following steps:

[0485] S2001, the PC's keyboard and mouse service module listens for keyboard and mouse events and mouse movement.

[0486] This step is the same as S1614 in principle, and you can refer to the description of S1614 for details, which will not be repeated here.

[0487] In S2002, the PC's keyboard and mouse service module determines whether the current mouse pointer has reached the edge of the screen based on the monitored keyboard and mouse movement.

[0488] The method for determining whether the mouse pointer has reached the edge of the screen can be found in the descriptions of the embodiments S1301-S1305 above, and will not be repeated here.

[0489] S2003: If the current mouse pointer reaches the edge of the screen, determine whether there is a device that can be passed through the current screen edge.

[0490] For example, when the left side of the PC is the tablet 13 and there is no connected device on the right side, if the current mouse pointer on the PC screen reaches the left edge of the screen and there is a device, the tablet 13, that can be passed through, then S2004 is executed; if the current mouse pointer on the PC screen reaches the right edge of the screen and there is no device that can be passed through, then there is no need to consider the keyboard and mouse crossing.

[0491] S2004: If there is a device that can pass through the edge of the current screen, determine whether the current mouse pointer belongs to the local keyboard and mouse (PC keyboard and mouse).

[0492] Depend on Figure 17 and Figure 18 As shown in the application scenario, the mouse pointer on the PC screen may be the mouse pointer of the PC's own keyboard and mouse, or it may be a mouse pointer that has been transferred from an Android device.

[0493] If the current mouse pointer belongs to the local keyboard and mouse (PC's keyboard and mouse), then... Figure 9 The scenario shown depicts the keyboard and mouse from a PC being transferred to an Android device. For instructions on how to transfer the keyboard and mouse, please refer to [link / reference needed]. Figure 12 Steps S1205-S1215 in the embodiment will not be repeated here.

[0494] It should be noted that, Figure 20 The diagram illustrates the scenario where the mouse pointer moves from the left side of the PC screen to Android device I; correspondingly, steps S1205-S1215 are executed between the PC and Android device I. It can be understood that if the mouse pointer moves from the right side of the PC screen to Android device II, steps S1205-S1215 are executed between the PC and Android device II.

[0495] If the current mouse pointer does not belong to the local keyboard and mouse (PC's keyboard and mouse), then proceed to step S2005.

[0496] S2005: If the current mouse pointer does not belong to the local keyboard and mouse (PC keyboard and mouse), then determine whether the current mouse pointer belongs to the receiving end.

[0497] Here, the receiving end refers to the Android device corresponding to the current traversing direction. For example, such as... Figure 1 In the communication system shown, if the current mouse pointer moves left across the screen of computer 11, and the left side of computer 11 is tablet 13, then tablet 13 is the receiving end. If the current mouse pointer moves right across the screen of computer 11, and the right side of computer 11 is mobile phone 12, then mobile phone 12 is the receiving end.

[0498] If the current mouse pointer belongs to the receiving end, that is, the receiving end is the source end, then... Figure 14 The example shown illustrates how the keyboard and mouse on an Android device can be transferred from a PC back to the Android device. For instructions on how to transfer the keyboard and mouse, please refer to [link / reference needed]. Figure 16 Steps S1616-S1620 in the embodiment will not be described again here.

[0499] Here, "source" refers to the Android device to which the current mouse pointer belongs. For example, such as... Figure 1 In the communication system shown, if the current mouse pointer belongs to mobile phone 12, then mobile phone 12 is the source, indicating that the keyboard and mouse of mobile phone 12 first travel to computer 11, and then from computer 11 to tablet 13. If the current mouse pointer belongs to tablet 13, then tablet 13 is the source, indicating that the keyboard and mouse of tablet 13 first travel to computer 11, and then from computer 11 back to tablet 13.

[0500] If the current mouse pointer does not belong to the receiving end, it means that the current mouse pointer has crossed over from Android device II, then S2006 is executed.

[0501] It should be noted that, Figure 20 The illustration shows a scenario where the mouse pointer moves from the left side of the PC screen to Android device I, where Android device I is the receiving end and Android device II is the source end.

[0502] S2006, if the current mouse pointer does not belong to the receiving end (Android device I), then send a transfer instruction (first instruction) to the source end (Android device II).

[0503] Correspondingly, the source end can receive the transfer instruction. This transfer instruction is used to inform the source end that the receiver of keyboard and mouse events has changed.

[0504] This step represents a scenario where the keyboard and mouse are moved from one Android device to another via a PC. For example, as... Figure 1 In the communication system shown, if the current mouse pointer on the computer 11 screen belongs to the keyboard and mouse of the mobile phone 12 (source end), and the current mouse pointer moves to the left to the tablet 13 (receiving end), it is equivalent to the receiving end that the keyboard and mouse crosses from the computer 11 to the tablet 13.

[0505] S2007, after receiving the transfer instruction, the keyboard and mouse service module of the source end (Android device II) sends a traversal instruction to the keyboard and mouse service module of the receiving end (Android device I).

[0506] In some implementations, the source end can send a traversal command to the receiving end via a PC. That is, the source end first sends the traversal command to the PC, and then the PC forwards it to the receiving end.

[0507] In other implementations, the source end can request to establish a keyboard and mouse connection directly with the receiving end. After the keyboard and mouse connection is established, the source end directly sends a traversal command to the receiving end.

[0508] S2008, after receiving the traversal command, if the current traversal status is not traversal, the receiving end (Android device I) replies with confirmation information to the keyboard and mouse service module of the source end (Android device II).

[0509] In some scenarios, if the receiving end is currently in a traversal state, it will reply with occupancy information to the keyboard and mouse service module of the source end to inform the source end that the receiving end is currently in a traversal state. In this case, the source end can notify the user via the screen that keyboard and mouse traversal is not possible at this time.

[0510] It should be noted that in this embodiment of the application, if a device is currently in a traversal state, it will no longer traverse between keyboard and mouse devices.

[0511] S2009, the keyboard and mouse service module of the receiving end (Android device I) updates the traversal status based on the traversal command.

[0512] In this step, updating the crossing status means changing the crossing status from not crossing to crossing in progress.

[0513] S2010, the keyboard and mouse service module of the receiving end (Android device I) determines the crossing position based on the crossing command.

[0514] In this step, the method for the receiving end to determine the crossing location can be found in the description of embodiments S1301-S1305.

[0515] S2011, after receiving the confirmation information, the keyboard and mouse service module of the source end (Android device II) packages the keyboard and mouse events reported by the input frame of the source end (Android device II) according to the preset protocol to obtain the keyboard and mouse event data packet.

[0516] S2012, the source end (Android device II) sends the data packets of keyboard and mouse events to the keyboard and mouse service module of the receiving end (Android device I).

[0517] In this embodiment, the communication system uses a PC as the communication center; in other words, all interactions between Android device I and Android device II must go through the PC. One implementation of this step is as follows: the source-end keyboard and mouse service module sends the keyboard and mouse event data packets to the source-end communication module; the source-end communication module sends the data packets to the PC's communication module; the PC's communication module forwards the data packets to the receiving-end communication module; and the receiving-end communication module sends the data packets to the receiving-end keyboard and mouse service module.

[0518] To improve communication efficiency, another implementation of step S2012 is as follows: the source and the receiving end establish a communication connection through a PC; after establishing the communication connection between the source and the receiving end, the keyboard and mouse service module of the source end sends the data packets of keyboard and mouse events to the communication module of the source end, and the communication module of the source end sends the data packets to the communication module of the receiving end; the communication module of the receiving end sends the data packets to the keyboard and mouse service module of the receiving end.

[0519] It should be noted that, in order to simplify the steps in the diagram, Figure 20 The above data interaction process is omitted in S2012.

[0520] S2013, the keyboard and mouse service module of the receiving end (Android device I) responds to the keyboard and mouse event.

[0521] S2014, The keyboard and mouse service module of the receiving end (Android device I) listens for keyboard and mouse crossover.

[0522] Steps S2013-S2014 are based on the same principle as S1213-S1214. For details, please refer to the description of S1213-S1214, which will not be repeated here.

[0523] In some scenarios, communication failures may occur between the source and receiver, causing a communication breakdown. In this scenario, S2015 will be executed.

[0524] S2015, If the communication between the source and the receiver is interrupted, the source and the receiver shall respectively restore their states.

[0525] The source-end recovery state includes the source stopping sending data packets of keyboard and mouse events to the receiving end, restoring the mouse pointer of the source keyboard and mouse to a first preset position on the source screen, and maintaining the communication connection with the PC. For example, the first preset position could be the center of the source screen, or the initial position on the source screen where the mouse event occurred. The receiving end recovery state includes restoring the mouse pointer of the receiving end keyboard and mouse to a second preset position on the receiving end screen, and maintaining the communication connection with the PC. For example, the second preset position could be the center of the receiving end screen, or the position of the mouse pointer on the receiving end screen before the keyboard and mouse event occurred. In other words, after the recovery state, the mouse pointers of the source and receiving ends return to their respective screens.

[0526] It should be noted that, in order to simplify the steps, Figure 20 The diagram only shows a portion of the software architecture in PCs and Android devices. This does not mean that other modules not shown do not participate in keyboard and mouse traversal. Modules participating in keyboard and mouse traversal can be found in the descriptions of other embodiments.

[0527] It should be noted that, Figure 20 The example shows a scenario where Android device II is the source and Android device I or PC is the receiver. The principle is similar for the scenario where Android device I is the source and Android device II or PC is the receiver, so it will not be described again here.

[0528] For example, in some application scenarios, Figure 20 In this embodiment, the PC can be a second device, Android device I can be a first device, and Android device II can be a third device; or, the PC can be a second device, Android device II can be a first device, and Android device I can be a third device. In this application scenario, the second device is communicatively connected to both the first device and the third device.

[0529] In this embodiment, in application scenarios where keyboard and mouse crossing occurs between multiple terminal devices, when the mouse pointer crosses from the PC screen to the Android device I screen, the PC side determines whether it is the local keyboard and mouse. This allows it to distinguish whether the crossing is from the PC to Android device I, from the PC back to Android device I, or from Android device II to Android device I. This method distinguishes between scenarios involving keyboard and mouse crossing across multiple terminal devices, preventing confusion in the crossing status and ensuring the reliability of keyboard and mouse crossing.

[0530] In this embodiment, after the mouse pointer crosses a path, the crossed mouse pointer and the local mouse pointer can be combined into a single mouse pointer. This avoids mouse pointer confusion.

[0531] It should be noted that the above embodiments are all described with the PC as the communication center. In practical applications, an Android device can also be used as the center, that is, the Android device connects to multiple terminal devices, and the Android device initiates the keyboard and mouse connection. In other words, the Android device is the initiator and the PC is the receiver. The principle of the keyboard and mouse traversal method with the Android device as the communication center is the same as that with the PC as the communication center, and will not be described again in the embodiments of this application.

[0532] The various embodiments described herein can be independent solutions or combinations thereof based on their inherent logic, and all such solutions fall within the protection scope of this application.

[0533] The foregoing mainly describes the solutions provided by the embodiments of this application from the perspective of method steps. It is understood that, in order to achieve the above functions, the electronic device implementing this method includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of protection of this application.

[0534] This application also provides a chip coupled to a memory, which is used to read and execute computer programs or instructions stored in the memory to perform the methods in the above embodiments.

[0535] This application also provides an electronic device including a chip for reading and executing computer programs or instructions stored in a memory, causing the methods in the various embodiments to be performed.

[0536] This embodiment also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the methods described in the above embodiments.

[0537] This embodiment also provides a computer program product, which is a computer-readable storage medium storing program code. When the computer program product is run on a computer, it causes the computer to perform the above-described related steps to implement the method in the above embodiment.

[0538] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the chip to execute the gaze point estimation method in the above-described method embodiments.

[0539] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0540] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0541] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for bypassing keyboard and mouse restrictions, characterized in that, The method includes: When the cursor of the input device is displayed on the screen of the first device, the first device detects a first operation performed on the input device. The first operation is used to move the cursor displayed on the screen of the first device in a first direction, wherein the input device is used to input control commands. In response to the first operation, the first device controls the cursor to move out of the screen of the first device and into the screen of the second device for display, wherein the second device is located in the first direction of the first device and establishes a connection with the first device; The second device detects a second operation performed on the input device, the second operation being used to move a cursor displayed on the screen of the second device in a second direction; In response to the second operation, the second device controls the cursor to move out of the screen of the second device and into the screen of the third device for display, wherein the third device is located in the second direction of the second device and is connected to the second device; If the input device is connected to the first device, in response to the fourth operation, the second device determines whether the first device and the third device are the same. The fourth operation is used to move the cursor displayed on the screen of the second device in the second direction. If the first device and the third device are the same, the first device updates the crossing status of the first device to "not crossed". The second device updates the crossing status of the second device to "not crossed".

2. The method as described in claim 1, characterized in that, In response to the first operation, the first device controls the cursor to move out of the screen of the first device and into the screen of the second device for display, including: In response to the first operation, the first device locates a second device that has established a connection with the first device in the first direction; The first device controls the cursor to move out of the screen of the first device and into the screen of the second device for display.

3. The method as described in claim 1 or 2, characterized in that, The method further includes: In response to the first operation, the first device determines whether the input device is connected to the first device; If the input device is connected to the first device, the first device updates the traversal status of the first device to "traversing". The second device updates its traversal status to "traversing".

4. The method as described in claim 3, characterized in that, After the first device determines whether the input device is connected to the first device in response to the first operation, the method further includes: If the input device is connected to the first device, the first device detects a third operation, the third operation being used to input the control command; The first device intercepts the first input event corresponding to the third operation; The first device sends the first input event to the second device; After receiving the first input event, the second device executes the control command corresponding to the first input event.

5. The method according to any one of claims 1 to 4, characterized in that, In response to the second operation, the second device controls the cursor to move out of the screen of the second device and onto the screen of the third device for display, including: In response to the second operation, the second device locates a third device that has established a connection with the second device in the second direction; The second device controls the cursor to move out of the screen of the second device and into the screen of the third device for display.

6. The method as described in claim 1, characterized in that, In response to the fourth operation, after the second device determines whether the first device and the third device are the same, the method further includes: If the first device is the same as the third device, the first device stops intercepting the first input event corresponding to the third operation, and the third operation is used to input the control command. The first device executes the control command corresponding to the first input event.

7. The method as described in claim 1, characterized in that, In response to the fourth operation, after the second device determines whether the first device and the third device are the same, the method further includes: If the first device is the same as the third device, the first device controls the cursor to be displayed at a first preset position on the screen of the first device.

8. The method as described in claim 1, characterized in that, In response to the fourth operation, after the second device determines whether the first device and the third device are the same, the method further includes: If the first device is different from the third device, the second device sends a first instruction to the first device; After receiving the first instruction, the first device controls the cursor to move out of the screen of the second device and into the screen of the third device for display.

9. The method as described in claim 8, characterized in that, After receiving the first instruction, the first device controls the cursor to move out of the screen of the second device and onto the screen of the third device for display, including: After receiving the first instruction, the first device establishes a connection with the third device; After the first device establishes a connection with the third device, the first device controls the cursor to move out of the screen of the second device and into the screen of the third device for display.

10. The method as described in claim 9, characterized in that, After the first device and the third device establish a connection, the method further includes: The second device updates its crossing status to "not crossed"; The third device updates its traversal status to "traversing".

11. The method as described in claim 9, characterized in that, After the first device and the third device establish a connection, the method further includes: If the first device loses connection with the third device, the first device controls the cursor to be displayed at a first preset position on the screen of the first device; The first device maintains a communication connection with the second device.

12. The method as described in claim 9, characterized in that, After the first device and the third device establish a connection, the method further includes: If the first device loses connection with the third device, the first device stops intercepting the input event corresponding to the third operation, which is used to input the control command. The first device executes the control command corresponding to the input event.

13. The method as described in claim 3, characterized in that, After the first device determines whether the input device is connected to the first device in response to the first operation, the method further includes: If the input device is not connected to the first device, the first device updates the traversal status of the first device to "not traversed".

14. The method as described in claim 13, characterized in that, After the first device determines whether the input device is connected to the first device in response to the first operation, the method further includes: If the input device is not connected to the first device, the first device sends a second instruction to the second device; After receiving the second instruction, the second device determines whether the input device is connected to the second device; If the input device is connected to the second device, the second device controls the cursor to move out of the screen of the first device and display it at a second preset position on the screen of the second device.

15. The method as described in claim 14, characterized in that, After determining whether the input device is connected to the second device, the method further includes: If the input device is connected to the second device, the second device detects a fifth operation, which is used to input the control command. The second device stops intercepting the second input event corresponding to the fifth operation; The second device executes the control command corresponding to the second input event.

16. The method as described in claim 14, characterized in that, After determining whether the input device is connected to the second device, the method further includes: If the input device is connected to the second device, the second device updates the traversal status of the second device to "not traversed".

17. The method as described in claim 14, characterized in that, After determining whether the input device is connected to the second device, the method further includes: If the input device is connected to the second device, in response to the sixth operation, the second device determines a first crossing position on the screen of the second device, the sixth operation being used to move the cursor displayed on the screen of the second device in the second direction; The second device sends the first crossing location to the third device; The third device determines the second crossing position on its screen based on the first crossing position; The second device controls the cursor to move off the screen of the second device from the first crossing position; The third device controls the cursor to enter the screen of the third device from the second crossing position.

18. The method as described in claim 17, characterized in that, Following the sixth operation, the method further includes: The second device updates its traversal status to "traversing".

19. The method as described in claim 17, characterized in that, Following the sixth operation, the method further includes: The second device detects the seventh operation, which is used to input the control command; The second device intercepts the third input event corresponding to the seventh operation; The second device sends the third input event to the third device; After receiving the third input event, the third device executes the control command corresponding to the third input event.

20. The method as described in claim 14, characterized in that, After determining whether the input device is connected to the second device, the method further includes: If the input device is not connected to the second device, the second device determines a fourth device connected to the input device; The second device sends a third instruction to the fourth device; After receiving the third instruction, the fourth device controls the cursor to move out of the screen of the first device and into the screen of the second device.

21. The method according to any one of claims 1 to 20, characterized in that, In response to the first operation, the first device controls the cursor to move out of the screen of the first device and into the screen of the second device for display, including: In response to the first operation, the first device determines whether the cursor on the screen of the first device has reached the edge of the screen; If the cursor on the screen of the first device reaches the edge of the screen, the first device controls the cursor to move out of the screen of the first device and into the screen of the second device for display.

22. A communication system, characterized in that, The communication system includes a first device, a second device, and a third device; The second device is connected to both the first device and the third device. The first device, the second device, and the third device in the communication system are used to perform the method as described in any one of claims 1 to 21.

23. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 21.

Images