Concurrent method, device and storage medium for screen collaboration and keyboard and mouse sharing
By adding MagicLink channel connection function to the screen collaboration service, the problem of screen collaboration and keyboard and mouse sharing cannot be used concurrently, and concurrent operations between multiple devices are achieved and user experience is improved.
Patent Information
- Application Number
- CN202310848788.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-11
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2043-07-11
AI Technical Summary
In the prior art, screen collaboration and keyboard and mouse sharing functions cannot be used concurrently, resulting in the lack of support for keyboard and mouse sharing connections in screen collaboration scenarios, and vice versa, affecting the user's multi-device office and entertainment experience.
By adding functional modules that support MagicLink channel connection in the screen collaboration service, it allows the same data transmission unit to establish screen collaboration connection in the keyboard and mouse sharing scenario, realizing the concurrent use of screen collaboration and keyboard and mouse sharing functions.
The concurrent use of screen collaboration and key-and-mouse sharing functions is realized. Users can easily perform screen collaboration and key-and-mouse operations between multiple devices without replacing the key-and-mouse device, improving the user experience.
Smart Images

Figure CN117707450B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of electronic devices, and in particular to a concurrent method, device and storage medium for screen collaboration and keyboard and mouse sharing. Background Art
[0002] The so-called screen collaboration is to connect devices of different systems collaboratively through distributed technology, and then use the Sink end (receiving end) as an extended screen of the Source end (sending end), so as to achieve a multi-screen office experience. The so-called keyboard and mouse sharing means that multiple electronic devices that establish a keyboard and mouse sharing connection can share a set of input / output (I / O) devices such as keyboards and mice.
[0003] However, currently screen collaboration and keyboard and mouse sharing cannot be used concurrently (at the same time), that is, in the screen collaboration scenario, keyboard and mouse sharing connection is not supported. Accordingly, in the keyboard and mouse sharing scenario, screen collaboration connection is not supported. Summary of the invention
[0004] In order to solve the above technical problems, the present application provides a concurrent method, device and storage medium for screen collaboration and keyboard and mouse sharing, aiming to enable screen collaboration and keyboard and mouse sharing to be used concurrently without affecting each other.
[0005] In the first aspect, the present application provides a concurrent method for screen collaboration and keyboard and mouse sharing. The method is applied to a system including a first electronic device and a second electronic device, including: the first electronic device searches for the second electronic device based on the first connection technology in response to the operation of establishing a screen collaborative connection; when the second electronic device is searched based on the first connection technology, the first electronic device determines whether the second electronic device can be searched through the second connection technology; when the second electronic device is searched based on the second connection technology, the first electronic device establishes a screen collaborative connection with the second electronic device based on the second connection technology; wherein, based on the second connection technology, the first electronic device can also establish a keyboard and mouse sharing connection with the second electronic device; when the second electronic device is not searched based on the second connection technology, the first electronic device establishes a screen collaborative connection with the second electronic device based on the first connection technology; wherein the first connection technology and the second connection technology are two mutually exclusive connection technologies, and at the same time, the first electronic device can only establish a communication connection with the second electronic device based on the first connection technology or the second connection technology, and the communication connection includes a screen collaborative connection and a keyboard and mouse sharing connection.
[0006] The operating systems of the first electronic device and the second electronic device may be the same or different.
[0007] Exemplarily, in some possible situations, the first electronic device may be a Windows system, such as a PC of Windows system.
[0008] Exemplarily, in some possible cases, the second electronic device may be an Android system, such as an Android system pad, a foldable screen mobile phone, etc.
[0009] Specifically in practical applications, the first electronic device can be regarded as the Source end mentioned in the following embodiments, and the second electronic device can be regarded as the Sink end in the following embodiments.
[0010] The first connection technology is a transmission technology implemented by NFC, such as the connection transmission service mentioned in the following embodiments.
[0011] Correspondingly, the screen cooperative connection between the first electronic device and the second electronic device established based on the first connection technology is such as the oneHop connection mentioned below. For the functional modules involved in the screen cooperative connection between the first electronic device and the second electronic device established based on the first connection technology, and the interaction logic between the functional modules, please refer to Figure 4 ,as well as Figure 4 The embodiments shown will not be described in detail here.
[0012] Among them, the second connection technology is a connection technology based on an identity authentication system that can achieve trusted interconnection across systems and devices, such as the data transmission unit implemented in the interconnection module in the following embodiments.
[0013] Correspondingly, the screen cooperative connection between the first electronic device and the second electronic device established based on the second connection technology is such as the MagicLink connection mentioned below. For the functional modules involved in the screen cooperative connection between the first electronic device and the second electronic device established based on the second connection technology, and the interaction logic between the functional modules, please refer to Figure 4 ,as well as Figure 4 The embodiments shown will not be described in detail here.
[0014] In addition, for the above-mentioned operation of triggering the establishment of screen collaborative connection, please refer to the following embodiment. Figure 4 , Figure 5A , Figure 5B , and the embodiments of these drawings will not be described in detail here.
[0015] In addition, for specific implementation details of the first electronic device determining whether the second electronic device can be searched through the second connection technology, see the following embodiment. Figure 4 ,as well as Figure 4The processing part of querying the online status of the interconnected device shown in the illustrated embodiment will not be repeated here.
[0016] Therefore, by determining whether the second electronic device searched based on the first connection technology can be searched through the second connection technology, and when the second electronic device can be searched through the second connection technology, a screen collaborative connection between the first electronic device and the second electronic device is established by adopting the second connection technology that can establish both a screen collaborative connection between the two electronic devices and a keyboard and mouse sharing connection between the two, thereby allowing the screen collaborative function and the keyboard and mouse sharing function to be used concurrently without affecting each other.
[0017] According to the first aspect, when the second electronic device is searched based on the first connection technology, the method also includes: the first electronic device determines whether the second electronic device supports the establishment of a screen collaborative connection using the second connection technology; when it is determined that the second electronic device supports the establishment of a screen collaborative connection using the second connection technology, the first electronic device executes the step of establishing a screen collaborative connection with the second electronic device based on the second connection technology; when it is determined that the second electronic device does not support the establishment of a screen collaborative connection using the second connection technology, the first electronic device executes the step of establishing a screen collaborative connection with the second electronic device based on the first connection technology.
[0018] For specific implementation details of the first electronic device determining whether the second electronic device supports the use of the second connection technology to establish a screen cooperative connection, see the following embodiment. Figure 4 ,as well as Figure 4 The processing part of querying whether the peer version matches shown in the illustrated embodiment will not be repeated here.
[0019] Therefore, when the second electronic device can be searched through the second connection technology, it is further considered whether the second electronic device supports the use of the second connection technology to establish a screen collaborative connection with the first electronic device, so that according to the judgment result, a suitable connection technology can be selected according to the actual situation to establish a screen collaborative connection between the first electronic device and the second electronic device.
[0020] According to the first aspect, or any implementation method of the first aspect above, the first electronic device determines whether the second electronic device supports the establishment of a screen collaborative connection using the second connection technology, including: the first electronic device obtains the version number supported by the second electronic device; the first electronic device determines whether the version number supported by the second electronic device is the same as the version number supported by the first electronic device; wherein the version number supported by the first electronic device and the version number supported in the second electronic device are both version numbers of functional modules implemented based on the second connection technology; when it is determined that the version number supported by the second electronic device is the same as the version number supported by the first electronic device, it is determined that the second electronic device supports the establishment of a screen collaborative connection using the second connection technology; when it is determined that the version number supported by the second electronic device is different from the version number supported by the first electronic device, it is determined that the second electronic device does not support the establishment of a screen collaborative connection using the second connection technology.
[0021] The functional module mentioned above is, for example, a data transmission unit implemented in the interconnection module mentioned in the following embodiments.
[0022] For specific implementation details of the first electronic device determining whether the second electronic device supports the use of the second connection technology to establish a screen cooperative connection, see the following embodiment. Figure 4 ,as well as Figure 4 The processing part of querying whether the peer version matches shown in the illustrated embodiment will not be repeated here.
[0023] Since the data transmission unit of the same version can ensure that the two establish a screen collaborative connection based on the second connection technology. Therefore, by determining whether the first electronic device and the second electronic device use the same version of the data transmission unit, it can be determined whether to use the first connection technology to establish a screen collaborative connection or the second connection technology to establish a screen collaborative connection, so as to better suit the actual application scenario.
[0024] According to the first aspect, or any implementation method of the first aspect above, the first electronic device determines whether the second electronic device supports the establishment of a screen collaborative connection using the second connection technology, including: the first electronic device obtains the version number supported by the second electronic device; the first electronic device determines whether the version number supported by the second electronic device is the same as the preset version number; wherein the version number supported by the second electronic device and the preset version number are both version numbers of functional modules implemented based on the second connection technology; when it is determined that the version number supported by the second electronic device is the same as the preset version number, it is determined that the second electronic device supports the establishment of a screen collaborative connection using the second connection technology; when it is determined that the version number supported by the second electronic device is not the same as the preset version number, it is determined that the second electronic device does not support the establishment of a screen collaborative connection using the second connection technology.
[0025] The functional module mentioned above is, for example, a data transmission unit implemented in the interconnection module mentioned in the following embodiments.
[0026] For specific implementation details of the first electronic device determining whether the second electronic device supports the use of the second connection technology to establish a screen cooperative connection, see the following embodiment. Figure 4 ,as well as Figure 4 The processing part of querying whether the peer version matches shown in the illustrated embodiment will not be repeated here.
[0027] Since the newly released version notification will add new functions based on the original version, by determining whether the version number of the data transmission unit of the second electronic device is a preset data transmission unit that can realize the establishment of a screen collaborative connection using the second connection technology, or is greater than the set version number, it can be determined whether to use the first connection technology to establish a screen collaborative connection, or the second connection technology to establish a screen collaborative connection, so as to better suit actual application scenarios.
[0028] According to the first aspect, or any implementation of the first aspect above, the version number supported by the second electronic device is written into the functional module integrating the second connection technology by the screen collaboration service of the second electronic device when the second electronic device is powered on.
[0029] The functional module mentioned above is, for example, the device management unit in the interconnection module of the Android device mentioned in the following embodiment.
[0030] Regarding the processing of writing the version number currently supported by the data transmission unit through the screen collaborative service item device management unit when the Android device or the Sink end is started, please refer to the embodiment Figure 4 ,as well as Figure 4 The embodiments shown will not be described in detail here.
[0031] According to the first aspect, or any implementation of the first aspect above, when the second electronic device is searched for based on the second connection technology, after the first electronic device establishes a screen collaborative connection with the second electronic device based on the second connection technology, the method also includes: based on the second connection technology, the first electronic device establishes a keyboard and mouse sharing connection with the second electronic device.
[0032] Therefore, when the first electronic device and the second electronic device have established a screen collaborative connection based on the second connection technology, keyboard and mouse sharing between the two is established based on the second connection technology, thereby realizing concurrent use of the screen collaborative function and the keyboard and mouse sharing function.
[0033] According to the first aspect, or any implementation of the first aspect above, in a scenario where screen collaboration and keyboard and mouse sharing occur concurrently, and a floating window serving as an extended screen of the first electronic device is displayed on the screen of the second electronic device, the method further includes: when the first electronic device detects that the mouse moves within the main screen of the first electronic device, main screen edge detection is started; wherein the mouse is connected to the first electronic device; when the main screen edge detection determines that the mouse has moved out of the main screen, the first electronic device sends a first mouse crossing request to the second electronic device, and the first mouse crossing request carries a first mouse offset; after receiving the first mouse crossing request, the second electronic device calculates the first mouse position according to the first mouse offset; when the first mouse position is outside the extended screen area, the second electronic device makes a first confirmation message for the first mouse crossing request, and sends the first confirmation message to the first electronic device; wherein the first confirmation message indicates that the mouse crosses from the main screen to the screen of the second electronic device; the first electronic device hides the first style mouse cursor displayed on the main screen, and based on the keyboard and mouse sharing function, displays the second style mouse cursor on the screen of the second electronic device.
[0034] The mouse mentioned in this aspect is an external mouse connected to the first electronic device, or a built-in touch pad.
[0035] For details on how the electronic device implements the main screen edge detection, see the following embodiments. Figure 6 , 7A to 7C , and the description of the embodiments corresponding to these figures will not be repeated here.
[0036] Among them, the first mouse crossing request is such as Figure 6 After the collaborative mode setting is completed in the embodiment, the keyboard and mouse sharing service of the Windows device detects the mouse movement for the first time and confirms the mouse crossing request sent to the Android device when the mouse exceeds the edge of the main screen.
[0037] The first confirmation message, for example, Figure 6 In the illustrated embodiment, the Android device sends a message to the Windows device for the first time to confirm that the mouse has crossed to the pad screen.
[0038] For details about the functional modules involved in the transmission process of the first mouse crossing request and the first confirmation message, as well as the specific interactions of the functional modules, please refer to Figure 6 ,as well as Figure 6 Parts of the illustrated embodiment will not be described in detail here.
[0039] The second style mouse cursor is displayed on the screen of the second electronic device, specifically, the second style mouse is closed and displayed at the first mouse position calculated above.
[0040] The first style of the mouse cursor is, for example, the arrow-style cursor 10a described in the following embodiment, and the second style of the mouse cursor is, for example, the dot-style cursor 10b described in the following embodiment.
[0041] Regarding the switching of the mouse in this aspect, for example Figure 7B The scene shown in (1) Figure 7B Switching of the mouse in the scene shown in (2).
[0042] According to the first aspect, or any implementation of the first aspect above, the method also includes: when the first mouse position is within the extended screen area, the second electronic device makes a second confirmation message for the first mouse crossing request, and sends the second confirmation message to the first electronic device; wherein the second confirmation message indicates that the mouse crosses from the main screen to the extended screen; the first electronic device hides the first style mouse cursor displayed on the main screen, and based on the screen collaboration function, displays the first style mouse cursor on the extended screen.
[0043] Understandably, this scenario is that the user quickly drags the mouse and moves the mouse to the extended screen directly after the mouse crosses. Figure 7B In the scene (1), we switch directly to Figure 7C The scene in (2).
[0044] According to the first aspect, or any implementation of the first aspect above, the method also includes: in the process of displaying a second style mouse cursor on the screen of the second electronic device, when the mouse continues to be moved, the first electronic device obtains the second mouse offset of the mouse, and sends the second mouse offset to the second electronic device; the second electronic device calculates the second mouse position according to the second mouse offset; when the second mouse position is within the extended screen area, the second electronic device makes a third confirmation message; wherein the third confirmation message indicates that the mouse passes from the screen of the second electronic device to the extended screen; the first electronic device replaces the second style mouse cursor with the first style mouse cursor, and based on the screen collaboration function, sets the first style mouse cursor to the extended screen.
[0045] Regarding the switching of the mouse in this aspect, for example Figure 7B The scene shown in (1) Figure 7B Switching of the mouse in the scene shown in (2).
[0046] For details on the implementation of this aspect, see Figure 6 , Figure 7B , and the parts of the embodiments corresponding to these figures will not be described in detail here.
[0047] According to the first aspect, or any implementation of the first aspect above, the method also includes: in the process of displaying a first style mouse cursor in the extended screen, when the mouse continues to be moved, the first electronic device starts extended screen edge detection; when it is determined through the extended screen edge detection that the mouse has moved out of the extended screen, the first electronic device sends a second mouse crossing request to the second electronic device, and the second mouse crossing request carries a third mouse offset; after receiving the second mouse crossing request, the second electronic device calculates the third mouse position according to the third mouse offset; when the third mouse position is outside the extended screen area, the second electronic device makes a fourth confirmation message for the second mouse crossing request, and sends the fourth confirmation message to the first electronic device; wherein the fourth confirmation message indicates that the mouse crosses from the extended screen to the screen of the second electronic device; the first electronic device hides the first style mouse cursor displayed on the extended screen, and based on the keyboard and mouse sharing function, displays the second style mouse cursor on the screen of the second electronic device.
[0048] Regarding the switching of the mouse in this aspect, for example Figure 7C The scene shown in (1) Figure 7C Switching of the mouse in the scene shown in (2).
[0049] For details on the implementation of this aspect, see Figure 6 , Figure 7C , and the parts of the embodiments corresponding to these figures will not be described in detail here.
[0050] In a second aspect, the present application provides an electronic device. The electronic device includes: a memory and a processor, the memory and the processor are coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes instructions of the method in the first aspect or any possible implementation of the first aspect.
[0051] The electronic device may be the first electronic device mentioned in the first aspect, or may be the second electronic device.
[0052] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.
[0053] In a third aspect, the present application provides a computer-readable medium for storing a computer program, wherein the computer program includes instructions for executing the method in the first aspect or any possible implementation of the first aspect.
[0054] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.
[0055] In a fourth aspect, the present application provides a computer program, comprising instructions for executing the method in the first aspect or any possible implementation of the first aspect.
[0056] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.
[0057] In a fifth aspect, the present application provides a chip, the chip comprising a processing circuit and a transceiver pin, wherein the transceiver pin and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the method in the first aspect or any possible implementation of the first aspect to control the receiving pin to receive a signal and control the sending pin to send a signal.
[0058] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.
[0059] In a sixth aspect, the present application provides a concurrent system for screen collaboration and keyboard and mouse sharing, which may include the first electronic device and the second electronic device involved in the above-mentioned first aspect or any possible implementation of the first aspect.
[0060] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] Figure 1A A schematic diagram of the hardware structure of an Android device is shown as an example;
[0062] Figure 1B A schematic diagram of the software architecture of an Android device is shown as an example;
[0063] Figure 2A The hardware structure diagram of a Windows device is shown as an example;
[0064] Figure 2B A schematic diagram of the software architecture of a Windows device is shown as an example;
[0065] Figure 3 A schematic diagram showing an exemplary Windows device and an Android device including interconnection modules;
[0066] Figure 4 The diagram is a schematic diagram of the interaction of the functional modules involved in the implementation process of the concurrent method of screen collaboration and keyboard and mouse sharing;
[0067] Figure 5A and Figure 5B A schematic diagram of a user interface for triggering device discovery is shown as an example;
[0068] Figure 6 The diagram is a schematic diagram showing the interaction of the functional modules involved in mouse switching in a scenario where the screen collaboration function and the keyboard and mouse sharing function are used concurrently;
[0069] 7A to 7C The figure is a schematic diagram showing the changes of the cursor on the Windows device screen, the extended screen, and the Android device screen during the mouse switching process. DETAILED DESCRIPTION
[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0071] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0072] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.
[0073] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0074] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.
[0075] With the popularization of electronic devices, more and more users own multiple (at least 2) electronic devices. In order to facilitate users to use multiple electronic devices for office and entertainment, screen collaboration function and keyboard and mouse sharing function are favored by more and more users.
[0076] Based on the screen collaboration function, users can achieve a multi-screen (at least 2 screens) experience. For example, for multiple files (documents, pictures, etc.) on the Source side, such as file A and file B, open file A on the Source side and file B on the Sink side, and achieve a tiled display of file A and file B, which is convenient for users to view at the same time.
[0077] Based on the keyboard and mouse sharing module, electronic devices such as mobile phones and tablets that do not have or are not connected to input / output (I / O) devices such as keyboards and mice can share the use of computer keyboards, mice or touchpads.
[0078] Although the screen collaboration function and keyboard and mouse sharing function greatly facilitate users to use multiple electronic devices for office and entertainment, the screen collaboration function is based on the connection transmission service (such as OneHop) to implement the call of the bluetooth (bluetooth, BT) chip or the wireless fidelity (wireless fidelity, WiFi) chip, and then establish a screen collaboration connection between the Source and the Sink. The keyboard and mouse sharing function is based on the data transmission unit (such as MagicLink) in the interconnection module (such as the MagicRing trust ring) to implement the call of the BT chip (hereinafter referred to as: BT) or the WiFi chip (hereinafter referred to as: WiFi). Therefore, when BT and WiFi are preempted by the connection transmission service, the data transmission unit cannot use BT and WiFi. Correspondingly, when BT and WiFi are preempted by the data transmission unit, the connection transmission service cannot use BT and WiFi.
[0079] In other words, the screen collaboration function and the keyboard and mouse sharing function cannot be used concurrently at present, that is, in the screen collaboration scenario, keyboard and mouse sharing connection is not supported. Correspondingly, in the keyboard and mouse sharing scenario, screen collaboration connection is not supported. This means that in the screen collaboration scenario, the user needs to connect different keyboards and mice to the Source and Sink ends respectively, and then process the displayed content on the Source and Sink ends by manipulating different keyboards and mice. In the keyboard and mouse sharing scenario, if you want to view a file stored on the Source end (such as file A) on the Sink end, you need to use the shared mouse to copy a copy of file A stored on the Source end and transfer it to the Sink end for storage (regarded as file A'), and then use the shared keyboard and mouse to open file A' stored on the Sink end. If the user edits the content in file A' on the Sink end, file A on the Source end cannot synchronize the modification.
[0080] In view of this, the present application provides a concurrent method for screen collaboration and keyboard and mouse sharing, which aims to enable screen collaboration and keyboard and mouse sharing to be used concurrently without affecting each other. Specifically in the present application, in order to enable the concurrent use of the screen collaboration function and the keyboard and mouse sharing function, support for MagicLink channel connection is added to the screen collaboration service corresponding to the screen collaboration business, that is, the ability to call the function module / service / interface in the interconnection module is added to the screen collaboration service corresponding to the screen collaboration business, so that the screen collaboration connection can also be realized through the data transmission unit in the interconnection module.
[0081] In this way, when the data transmission unit preempts the use of BT and WiFi, that is, in the keyboard and mouse sharing scenario, the screen collaborative connection can be established based on the same data transmission unit without the user intervening to disconnect the keyboard and mouse sharing connection in order to achieve the establishment of the screen collaborative connection.
[0082] Regarding the screen collaborative service mentioned above, it can also be directly expressed as a collaborative service in some possible situations. The embodiments of the present application are explained using the screen collaborative service as an example.
[0083] In order to better understand the technical solution provided by the embodiments of the present application, before describing the technical solution of the embodiments of the present application, the hardware structure and software architecture of the electronic device to which the embodiments of the present application are applicable are first described in conjunction with the accompanying drawings.
[0084] The electronic devices to which the embodiments of the present application can be applied include but are not limited to mobile phones (such as foldable screen mobile phones), tablets, PCs, etc., which are not listed one by one here and the present application does not impose any restrictions on this.
[0085] In addition, it should be noted that in the technical solution provided in the embodiments of the present application, any of the above-mentioned electronic devices can be used as a Source end or as a Sink end.
[0086] In addition, the operating system of the above electronic device includes but is not limited to Windows system, Android system and iOS system. That is to say, in the technical solution provided in the embodiment of the present application, the operating system of the source end and the sink end can be the same or different.
[0087] For the sake of convenience, the embodiment of the present application takes the Source end as a Windows device, such as a PC with a Windows operating system; the Sink end as an Android device, such as a tablet with an Android operating system as an example. It should be understood that this limitation is only an example listed to better illustrate the technical solution of this embodiment, and is not the only limitation to this embodiment.
[0088] See also Figure 1A , which illustrates an exemplary hardware structure of an Android device.
[0089] See also Figure 1A Exemplarily, the Android device may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.
[0090] Among them, the processor is used to execute program instructions, thereby realizing the technical solution of concurrent use of the screen collaboration function and the keyboard and mouse sharing function provided in each embodiment of the present application.
[0091] The audio module 170 may include a speaker 170A, a receiver 170B, a microphone 170C, an earphone jack 170D, and the like.
[0092] Among them, the sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc., which are not listed here one by one and the present application does not impose any limitation on this.
[0093] The USB interface may also include USB interfaces for a keyboard and a mouse.
[0094] Among them, BT and WLAN (specifically WiFi in this case) in the wireless communication module are specifically used to realize keyboard and mouse sharing connection and screen collaborative connection.
[0095] For the functions and uses of the above hardware included in the Android device, please refer to the hardware structure description documents of existing Android devices, which will not be repeated here.
[0096] Furthermore, it should be understood that Figure 1A The Android device shown is only an example. In a specific implementation, the Android device may have more or fewer components than those shown in the figure, may combine two or more components, or may have a different component configuration. Figure 1A The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.
[0097] See also Figure 1B , which illustrates an exemplary software architecture of an Android device.
[0098] Before explaining the software structure of Android devices, the architectures that can be adopted by software systems of Android devices and other operating systems, such as Windows systems, are first explained.
[0099] Specifically, in actual applications, whether it is a Windows device or an Android device, its software system (operating system) can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In addition, for Windows devices and Android devices, they can be divided into user state and kernel state.
[0100] like Figure 1B As shown in Figure 1, the layered architecture of Android devices divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some implementations, the Android system is divided into five layers, from top to bottom: the application layer (Application, APP layer), the framework layer (Application Framework, FWK layer), the Android runtime (Android runtime) and system library, the hardware abstraction layer (HAL layer), and the kernel layer (Linux Kernel, kernel layer).
[0101] Among them, the APP layer can include a series of application packages. Figure 1BAs shown, the application package may include camera, game, video, live broadcast and other applications, which are not listed here one by one and are not limited in this application.
[0102] Among them, the FWK layer provides application programming interfaces (APIs) and programming frameworks for applications in the APP layer. In some implementations, these programming interfaces and programming frameworks can be described as functions. Figure 1B As shown, the FWK layer may include connection transmission services, keyboard and mouse sharing services, screen collaboration services, interconnection modules, interconnected device authentication services, cross-device collaborative authentication services and other functions, which are not listed one by one here and are not limited in this application.
[0103] Among them, the connection transmission service is the connection technology used by the native screen collaborative service to establish a collaborative connection.
[0104] Among them, the keyboard and mouse sharing service is a service process that implements the keyboard and mouse sharing function. Specifically in each embodiment of the present application, the keyboard and mouse sharing service will also interact with the screen collaboration service and the data transmission unit in the interconnection module, thereby realizing the concurrent use of the keyboard and mouse sharing function and the screen collaboration function in this case. For details on the specific operations that the keyboard and mouse sharing service can perform, see the following Figure 4 , Figure 6 The description of the embodiment will not be repeated here.
[0105] Among them, the screen collaboration service is a service process that implements the screen collaboration function. Specifically in each embodiment of the present application, the screen collaboration service not only has the characteristics of the screen collaboration service in the existing solution, but also adds the transformation characteristics of MagicLink channel switching and compatibility processing. For details of the specific operations that the screen collaboration service can perform, please refer to the following Figure 4 , Figure 6 The description of the embodiment will not be repeated here.
[0106] The interconnection module can also be expressed as a trust ring or MagicRing trust ring. Specifically, it refers to a connection technology between different devices. This connection technology is based on an identity authentication system to achieve cross-system multi-device trusted interconnection, so that services can be intelligently transferred and shared between devices. Specifically in each embodiment of the present application, in order to realize the screen collaboration function and the keyboard and mouse sharing function, the interconnection module needs to include Figure 3 The device management unit (service / module / function), access networking unit, and data transmission unit are shown.
[0107] The device management unit is used to store and synchronize the service status and attributes of each service connected to the interconnection module.
[0108] Regarding the access networking unit, it is used to call BT or WiFi and discover trusted ring devices.
[0109] It should be noted that the trust ring device mentioned in this embodiment is an electronic device that also integrates the above-mentioned keyboard and mouse sharing service, screen collaboration service and interconnection module.
[0110] Regarding the data transmission unit, it is used to establish a MagicLink connection, and then realize data negotiation, transmission and other operations in keyboard and mouse sharing scenarios and screen collaboration scenarios.
[0111] The functions of the collaborative service and data transmission modules in Android devices are the same as those in Windows devices, and will not be repeated here.
[0112] about Figure 3 The functional modules / units disclosed in the interconnection module shown in the embodiment of the present invention are described in detail in the following description of the specific interaction and processing logic in the scheme for concurrent use of the screen collaboration function and the keyboard and mouse sharing function provided by the present application. Figure 4 , Figure 6 The description of the embodiment will not be repeated here.
[0113] In addition, it should be noted that Figure 3 What is shown are only the functional modules / units required for the interconnection module to implement this case. In actual applications, according to business needs, the interconnection module may also include a unified interconnection framework, adaptation virtualization services, audio virtualization services, etc., which are not listed here one by one, and this application does not impose any restrictions on this.
[0114] Continue to see Figure 1B , exemplarily, the interconnected device authentication service and the cross-device collaborative authentication service are used to authenticate the Sink device to ensure the security in the screen collaboration and keyboard and mouse sharing scenarios.
[0115] This is the end of the introduction to the FWK layer. Figure 1B Continue to introduce other layers of Android devices.
[0116] Continue to see Figure 1B For example, Android Runtime includes a core library and a virtual machine. Android Runtime is responsible for scheduling and management of the Android system.
[0117] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.
[0118] The APP layer and FWK layer run in the virtual machine. The virtual machine executes the java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
[0119] Continue to see Figure 1B For example, the system library may include multiple functional modules, such as a surface manager, a media library, a three-dimensional (3D) graphics processing library (such as OpenGL ES), a two-dimensional (2D) graphics engine (such as SGL), etc.
[0120] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports multiple commonly used audio and video format playback and recording, as well as static image files. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing.
[0121] It can be understood that the 2D graphics engine mentioned above is a drawing engine for 2D drawing.
[0122] Continue to see Figure 1B For example, the HAL layer is an interface layer between the operating system kernel (kernel layer) and the hardware circuit. Its purpose is to isolate the FWK layer from the kernel so that Android does not rely too much on the kernel, so that the development of FWK can be carried out without considering the driver.
[0123] Continue to see Figure 1B For example, the HAL layer may include various interfaces, such as audio and video interfaces, USB interfaces, Bluetooth interfaces, WiFi interfaces, etc., which are not listed here one by one and are not limited in this application.
[0124] Continue to see Figure 1B , exemplarily, the kernel layer in the Android system is a layer between hardware and software. The kernel layer may include various processes / threads, a power management module, and various drivers, such as a WiFi driver, a Bluetooth driver, and a kernel driver (Uinput driver) for simulating user input events.
[0125] This is the end of the introduction to the software architecture of Android devices. It is understandable that: Figure 1B and Figure 3The layers in the software Android device shown and the functional modules contained in each layer do not constitute a specific limitation on the Android device. In some possible implementations, the Android device may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in this application.
[0126] See also Figure 2A , which illustrates an exemplary hardware structure of a Windows device.
[0127] See also Figure 2A Exemplarily, the Windows device may include: a processor 210, an internal memory interface 220, an external memory 230, a wireless communication module 240, a mouse interface 250, a keyboard interface 260, a charging management module 270, a power management module 280, a battery 290, etc.
[0128] Among them, the processor is used to execute program instructions, thereby realizing the technical solution of concurrent use of the screen collaboration function and the keyboard and mouse sharing function provided in each embodiment of the present application.
[0129] The mouse interface and the keyboard interface may be the same USB interface, that is, the USB interface may be used as both a mouse interface and a keyboard interface.
[0130] Among them, BT and WLAN (specifically WiFi in this case) in the wireless communication module are specifically used to realize keyboard and mouse sharing connection and screen collaborative connection.
[0131] For the functions and uses of the above hardware included in the Windows device, please refer to the hardware structure description documents of existing Windows devices, which will not be repeated here.
[0132] Furthermore, it should be understood that Figure 2A The Windows device shown is only an example. In a specific implementation, the Windows device may have more or fewer components than those shown in the figure, may combine two or more components, or may have a different component configuration. Figure 2A The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.
[0133] See also Figure 2B , which illustrates an exemplary software architecture for a Windows device.
[0134] From the above description, we can know that the Windows system can be divided into user state and kernel state. Figure 2BAs shown, from the perspective of user state, from bottom to top, it can include NTDLL, service process, system process, environment subsystem, interconnection module, user process, etc. Specifically in this embodiment, the concurrent use of screen collaboration function and keyboard and mouse sharing function mainly depends on the interconnection module, as well as the keyboard and mouse sharing service, screen collaboration service, connection transmission service, interconnection device authentication service, and cross-device collaborative authentication service in the service process.
[0135] The functions of the interconnection module, keyboard and mouse sharing service, screen collaboration service, connection transmission service, interconnected device authentication service, and cross-device collaborative authentication service in Windows devices are the same as those in Android devices, and are not repeated here.
[0136] For the functions of other processes, systems, NTDLL, etc. in user mode, please refer to the introduction of the existing Windows system architecture, which will not be repeated here.
[0137] In addition, it should be noted that in some possible implementations, the user process may include a process corresponding to the computer manager application that provides the above-mentioned screen collaboration function and keyboard and mouse sharing function. Through the process corresponding to the computer manager (such as the service management process), the management of various services and the scheduling of various virtualized services can be achieved. For example, the scheduling of the keyboard and mouse sharing service and the screen collaboration service in the service process.
[0138] Continue to see Figure 2B , for example, from the perspective of kernel state, it can include hardware abstraction layer, kernel and device driver layer, window management and image drawing module, and Windows executive body from bottom to top. Specifically, in the technical solution provided in this embodiment, the screen collaboration function and keyboard and mouse sharing function need to rely on WiFi or BT to achieve the corresponding connection, so it is necessary to involve functional modules related to WiFi and BT, such as WiFi driver, Bluetooth driver, etc.
[0139] In addition, for Windows devices that do not have I / O devices, they also need to use USB interfaces, Bluetooth, or WiFi to connect external keyboards, mice, etc. Therefore, in addition to the functional modules related to WiFi and Bluetooth, they also need to involve functional modules related to USB, such as USB drivers.
[0140] For descriptions of the Windows executive body in kernel state, other functional modules involved in the kernel and device driver layer, window management and drawing modules, hardware abstraction layer, etc., please refer to the introduction to the existing Windows system architecture, which will not be repeated here.
[0141] This is the end of the introduction to the software architecture of Windows devices. It is understandable that: Figure 2B and Figure 3 The layers in the software Windows device and the functional modules contained in each layer shown do not constitute a specific limitation on the Windows device. In some possible implementations, the Windows device may include more or fewer layers than shown in the figure, and each layer may include more or fewer components, which is not limited in this application.
[0142] Based on the Windows device and Android device shown in the above exemplary hardware structure and software architecture, the following is combined Figure 4 The process of connecting the screen collaboration service to the interconnection module so that the screen collaboration function and the keyboard and mouse sharing function can be used concurrently is described.
[0143] In order to facilitate the distinction between the functional modules in the Windows device (as the Source end in this embodiment) and the Android device (as the Sink end in this embodiment), in the following embodiments, the screen collaboration service, connection transmission service, interconnection module in the Source end, and the device management unit, access networking unit, and data transmission unit in the interconnection module are respectively represented as Source screen collaboration service, Source connection transmission service, Source interconnection module, Source device management unit, Source access networking unit, and Source data transmission unit; the screen collaboration service, connection transmission service, interconnection module in the Sink end, and the device management unit, access networking unit, and data transmission unit in the interconnection module are respectively represented as Sink screen collaboration service, Sink connection transmission service, Sink interconnection module, Sink device management unit, Sink access networking unit, and Sink data transmission unit.
[0144] See also Figure 4 For example, when a user triggers a screen collaboration service at the Source end, the Source screen collaboration service, in response to the operation behavior, triggers the Source connection transmission service to find a Sink end that can currently establish a screen collaboration connection with the Source end.
[0145] It should be noted that the connection transmission service in this embodiment is a transmission technology implemented by NFC technology. Based on the characteristics of NFC, the Source end can automatically discover the Sink end nearby that also supports the connection transmission service.
[0146] Continue to see Figure 4For example, when the Source connection transmission service finds that there is a Sink end nearby with which a screen collaborative connection can be established, the currently discovered (searched) Sink end will be reported to the Source screen collaborative service.
[0147] Regarding the Source screen collaborative service in this embodiment sensing the active discovery operation triggered by the user, in some possible implementations, for example, the corresponding window (such as Figure 5A As shown in the figure, select the Screen Collaboration option under the Smart Connectivity directory, select "Extended Mode" in the Collaboration Mode displayed on the right side of the window, and then click the "Connect Now" control to trigger it.
[0148] It should be understood that the above description is only an example listed for a better understanding of the technical solution of this embodiment, and is not the only limitation to this embodiment. In actual applications, users can also trigger by setting shortcuts, such as touching the edge of a Windows device with the edge of an Android device. This application does not limit this.
[0149] In addition, after the Source connection transmission service reports the currently discovered Sink end to the Source screen collaboration service, in order to better meet user needs, the currently discovered Sink end that can establish a screen collaboration connection can be displayed on the Source end screen. Regarding the window of available Sink ends displayed on the Source end, for example Figure 5B shown.
[0150] like Figure 5B As shown, in some possible implementations, in order to ensure that the electronic device desired by the user can be found, the window may display the following Figure 5B The three prompt messages ① to ③ are shown in the figure.
[0151] Continue to see Figure 5B For example, in some possible implementations, the currently discovered Sink terminals displayed in the list of available devices can be displayed in descending order according to conditions such as most recent use or number of uses / frequency, so that users can quickly find frequently used Sink terminal devices.
[0152] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation to this embodiment.
[0153] Continue to see Figure 4 , for example, after the Source discovers an available device through the Source connection transmission service, if the user clicks the "connect" option corresponding to an available device, such as Figure 5BAfter selecting the "Connect" option corresponding to the "GDI" device, based on the concurrent solution of the screen collaboration function and the keyboard and mouse sharing function provided in the embodiment of the present application, when the Source screen collaboration service adds the modified features of MagicLink channel switching and compatibility processing, it will first query the online status of the interconnected device in the Source device management unit according to the device unique identifier corresponding to the available device (hereinafter represented as: interconnected device) reported by the Source connection transmission service.
[0154] Specifically, the online status of the interconnected device in this embodiment refers to whether the interconnected device can be discovered (searched) by the Source access networking unit in the Sourc end. If it can be discovered by the Source access networking unit, it can be confirmed that the screen system service of the interconnected device has also added the transformation features of MagicLink channel switching and compatibility processing, that is, the Source end can establish a screen collaborative connection with the interconnected device as the Sink end based on the data transmission unit, and use the MagicLink connection used by the keyboard and mouse sharing connection to establish a screen collaborative connection, thereby realizing the concurrent existence of the screen collaborative function and the keyboard and mouse sharing function.
[0155] Continue to see Figure 4 For example, if the Source device management unit fails to find the connected device in the path it manages (local, cloud) after receiving the instruction to query the online status of the connected device sent by the Source screen collaborative service, that is, the connected device is not online, the Source device management unit will trigger the Source access networking unit to discover the connected device. That is, the Source access networking unit will search for the Sink access networking unit in the attachment area, and match the device with the same device unique identifier as the above-mentioned connected device from the searched Sink end.
[0156] For example, if the Source access networking unit discovers the interconnected device, the online status of the interconnected device is online. Figure 4 Steps for subsequent processing.
[0157] For example, if the Source access networking unit does not find the interconnected device, in one possible case, the Source access networking unit may continue to search for the Sink access networking unit in the attachment area until the interconnected device is found, and feedback to the Source screen collaboration service that the interconnected device is online, and then follow Figure 4In another possible situation, when the Source access networking unit fails to search for the interconnected device within the set time, it will directly rely on the Source connection transmission service and the Sink connection transmission service of the interconnected device to establish a screen collaborative connection between the two. The logic of establishing a screen collaborative connection between the two through the Source connection transmission service and the Sink connection transmission service can be found in Figure 4 The processing flow of the connection type being connection type 2 is not repeated here.
[0158] This embodiment takes the case where the Source access networking unit searches for the Sink access networking unit of the interconnected device as an example.
[0159] In addition, it should be noted that different electronic devices may use different version numbers of the data transmission unit (MagicLink), and data transmission units with different version numbers may support the establishment of different connections. For example, for some versions, such as data transmission units below version 7.2, it may only support the establishment of keyboard and mouse sharing connections, but not screen collaborative connections. Therefore, in order to ensure that the Sourc end can establish a MagicLink connection with the Sink data transmission unit of the Sink end through the Source data transmission unit, and then realize the screen collaborative connection between the Source and Sink ends based on the MagicLink connection, the Source screen collaborative service will notify the Source device management unit to obtain the version number currently supported by the interconnected device when it learns that the interconnected device is online.
[0160] Continue to see Figure 4 For example, regarding the version number of the Sink data transmission unit in the interconnected device, in one possible case, for example, it can be obtained by the Sink screen collaborative service and written into the Sink device management unit after the device is powered on.
[0161] Continue to see Figure 4 , exemplarily, after the Source device management unit obtains the version number currently supported by the interconnected device from the Sink device management unit, in one possible case, it will determine whether the version number currently supported by the interconnected device matches the version number of the Source end, such as determining whether the version number of the Sink end is the same as the version number of the Source end. If they are the same, it is determined that the two match.
[0162] Exemplarily, in another possible case, it can also be determined whether the version number currently supported by the interconnected device is greater than the set version number, or belongs to any version number in the set version number set / list. If it is greater than the set version number, or belongs to any version number in the set version number set / list, it is determined that the two match.
[0163] Continue to see Figure 4 For example, when the version number supported by the interconnected device matches the version number supported by the Source end, the connection type determined by the Source device management unit is connection type 1, that is, the screen collaborative connection can also be implemented based on the MagicLink connection established by the data transmission unit. Otherwise, the connection type is determined to be connection type 2, that is, the screen collaborative connection does not support the MagicLink connection implementation established using the data transmission unit, and needs to be implemented based on the oneHop connection corresponding to the connection transmission service.
[0164] Continue to see Figure 4 For example, in the case of connection type 1, the Source screen collaborative service triggers the Source data transmission unit and the Sink data transmission unit of the interconnected device through the Source access networking unit, and establishes a MagicLink connection between the Source and the Sink based on the MagicLink transmission technology.
[0165] Continue to see Figure 4 , exemplarily, after the Source end and the Sink end establish a MagicLink connection, the Sink data transmission unit will pull up the Sink screen collaborative service.
[0166] Continue to see Figure 4 For example, after the Sink screen collaboration service is started, data negotiation will be performed with the Source end through the MagicLink connection established between the Sink data transmission unit and the Source data transmission unit (negotiation of relevant data for establishing a screen collaboration connection). For the data information that needs to be negotiated during the data negotiation between the Source end and the Sink end, please refer to the existing documents on screen collaboration, which will not be repeated here. In this embodiment, the negotiated data is specifically carried out through the MagicLink connection, not the oneHop connection.
[0167] Continue to see Figure 4 , exemplarily, after the Sink end completes data negotiation with the Source end through the MagicLink connection, the screen collaborative connection between the two ends will be formally established successfully.
[0168] Because, in the case of connection type 1, the screen collaboration connection uses the same connection as the keyboard and mouse sharing connection, that is, the MagicLink connection. Therefore, for the case of connection type 1, the Source and Sink can realize the concurrent use of the screen collaboration function and the keyboard and mouse sharing function, so that the Sink can be used as an external extension screen of the Source while the user can also use the keyboard and mouse of the Source to directly operate the Sink. Since the entire process does not require the replacement of the keyboard and mouse device, it can greatly facilitate user use and improve the user experience.
[0169] This concludes the introduction to the processing logic for connection type 1.
[0170] Continue to see Figure 4 For example, in the case of connection type 2, the Source screen collaboration service will establish a oneHop connection between the Source end and the Sink end through the Source connection transmission service and the Sink connection transmission service of the interconnected device based on the oneHop transmission technology.
[0171] Continue to see Figure 4 , exemplarily, after the oneHop connection is established between the Source and the Sink end, the Sink connection transmission service will pull up the Sink screen collaboration service.
[0172] Continue to see Figure 4 , exemplarily, after the Sink screen collaboration service is pulled up, data negotiation is performed with the Source end through the oneHop connection established by the Sink connection transmission service and the Source connection transmission service. For the data information that needs to be negotiated during the data negotiation between the Source end and the Sink end, please refer to the existing documents on screen collaboration, which will not be repeated here.
[0173] Continue to see Figure 4 , exemplarily, after the Sink end completes data negotiation with the Source end through the oneHop connection, the screen collaborative connection between the two ends will be formally established successfully.
[0174] Because, in the case of connection type 2, the screen collaboration connection uses a different connection from the keyboard and mouse sharing connection, that is, one is implemented through the oneHop connection, and the other is implemented through the MagicLink connection. Therefore, for the case of connection type 2, the Source and Sink ends cannot realize the concurrent use of the screen collaboration function and the keyboard and mouse sharing function.
[0175] This concludes the introduction to the processing logic for connection type 2.
[0176] Through the above Figure 4The description of the screen collaboration and keyboard and mouse sharing method provided by this application can realize that both the collaborative connection and the keyboard and mouse sharing connection are realized through the MagicLink connection. Based on this method, in the scenario where the screen collaboration function and the keyboard and mouse sharing function coexist, the switching process of the same set of keyboard and mouse is as follows Figure 6 shown.
[0177] It is understandable that the keyboard and mouse are paired for traversal, and in the case of mouse traversal, the keyboard input will also be switched to the screen that the mouse currently traverses to. Therefore, the following directly describes the mouse switching process.
[0178] See also Figure 6 , exemplarily illustrating a scenario where keyboard and mouse sharing and screen collaboration exist concurrently, that is, the screen collaboration connection and the keyboard and mouse sharing connection are both realized through a MagicLink connection established by a data transmission unit in the Source end and a data transmission unit in the Sink end.
[0179] In addition, it should be noted that through the above Figure 5A It can be known that the collaborative mode may include a mirror mode and an extended mode. Specifically in this embodiment, the collaborative mode selected by the user is the extended mode, and the window acting as the extended screen of the Windows device on the Android device (eg, pad) is a floating window as an example.
[0180] Continue to see Figure 6 , for example, in order to determine the switching of the mouse, after the Source screen collaborative service informs the Sink screen collaborative service through the Source data transmission unit and the Sink data transmission unit that the currently selected collaborative mode is the extended mode, the Sink screen collaborative service will inform the Sink keyboard and mouse sharing service that the current collaborative mode is the extended mode.
[0181] Since the content displayed in the extended screen in the form of a floating window on the Android device is also the content in the Windows device, therefore, when the cursor corresponding to the mouse is located on the screen of the Windows device (hereinafter referred to as the main screen) and the extended screen, the same cursor style can be used, such as Figure 7B and Figure 7C The cursor style on the Android device screen (outside the extended screen area) can be Figure 7B and Fig. 7A Dot-style cursor 10b.
[0182] Because the cursor corresponding to the mouse has different styles when displayed on different screens. The Android screen has both an extended screen for Windows devices and the screen area of the Android device itself. When the Sink keyboard and mouse sharing service learns that the current collaborative mode is extended mode, and the corresponding extended screen in extended mode is in floating window form, the Sink keyboard and mouse sharing service will read the size of the extended screen in floating window form currently displayed on the Android device screen. For example, the coordinates of the upper left corner vertex and the lower right corner vertex of the extended screen.
[0183] Continue to see Figure 6 ,Exemplarily, when the Source screen collaboration service sets the collaboration mode to the extended mode, it will notify the Source keyboard and mouse sharing service in addition to notifying the Sink screen collaboration service.
[0184] Continue to see Figure 6 , exemplarily, after the Source keyboard and mouse sharing service learns that the current collaborative mode is the extended mode, in order to monitor the switching of the mouse, it will set the main screen coordinates and the extended screen coordinates.
[0185] Regarding the coordinates of the main screen, you can set the coordinates of the upper left corner and the lower right corner of the main screen. Fig. 7A As shown in (1), when the size of the main screen is 1920*1080, the vertex coordinates of the upper left corner of the main screen can be set to (0, 0), and the vertex coordinates of the lower right corner can be set to (1920, 1080).
[0186] The coordinates of the extended screen can be determined by reading the size of the extended screen from the Sink keyboard and mouse sharing service. Fig. 7A In the case where the vertex coordinates of the upper left corner of the extended screen shown in (1) on the Android device screen are (x1, y1) and the vertex coordinates of the lower right corner are (x2, y2), the vertex coordinates of the upper left corner of the extended screen set by the Source keyboard and mouse sharing service (exStart, eyStart) can be (x1, y1), and the vertex coordinates of the lower right corner (exEnd, eyEnd) can be (x2, y2).
[0187] Continue to see Figure 6 , exemplarily, based on the above settings, when the mouse moves within the main screen of the Windows device, the Source keyboard and mouse sharing service senses the movement of the mouse and starts the main screen edge detection process. For example, the coordinate information of the cursor 10a on the main screen is obtained, and then the cursor 10a is determined to have moved to the edge of the main screen according to the coordinate information of the cursor 10a, for example Figure 7B The area with the X-axis coordinate of 1920 is shown in (1).
[0188] Continue to see Figure 6 , exemplarily, when the X-axis coordinate corresponding to the cursor 10a exceeds the X-axis coordinate of the side edge of the main screen, such as Figure 7B At 1920 shown in (1), it is determined that the cursor 10a exceeds the edge of the main screen, and the mouse will cross, that is, cross from the main screen of the Windows device to the pad screen.
[0189] Continue to see Figure 6 For example, when it is determined that the cursor 10a exceeds the edge of the main screen, the Source keyboard and mouse sharing service will initiate a mouse crossing request to the Sink keyboard and mouse sharing service. Figure 6 The mouse crossing request will be sent to the Source data transmission unit through the Source keyboard and mouse sharing service, and then sent to the Sink data transmission unit by the Source data transmission unit based on the MagicLink transmission technology, and finally sent to the Sink keyboard and mouse sharing service by the Sink data transmission unit.
[0190] Continue to see Figure 6 , exemplarily, the mouse crossing request will carry the coordinate information of the cursor 10a, and the set extended screen coordinates (exStart, eyStart) and (exEnd, eyEnd) correspond to the (x1, y1) and (x2, y2) of the extended screen displayed on the pad screen, respectively. Therefore, based on the above coordinate information, the Sink keyboard and mouse sharing service can calculate the position of the cursor corresponding to the mouse to be displayed on the pad screen (hereinafter referred to as: mouse position). Since the size of the extended screen displayed in the form of a floating window on the pad is already determined, based on the currently determined mouse position, it can be determined whether the mouse has moved into or outside the extended screen area.
[0191] For the sake of convenience, this embodiment takes the case where the mouse is outside the extended screen area when the mouse crosses for the first time as an example. Figure 6 For example, when it is determined that the mouse is outside the extended screen area displayed on the pad, the Sink keyboard and mouse sharing service will inform the Source keyboard and mouse sharing service through the Sink data transmission unit and the Source data transmission unit that the current scene mouse has crossed to the pad screen, that is, Figure 6 The confirmation mouse crosses to the pad screen as shown in Figure 7B (1) becomes Figure 7B Chinese (2).
[0192] From the above description of the cursor styles on the main screen, extended screen, and pad screen, we can see that when the mouse crosses to the pad screen, the cursor corresponding to the mouse will change from Figure 7B(1) The cursor 10a displayed on the screen of the Windows device switches to Figure 7B (2) The cursor 10b is displayed on the pad screen.
[0193] pass Figure 7B Zhong (1) and Figure 7B As can be seen from the switching of the mouse shown in (2), when the mouse passes from the main screen to the pad screen, the Source keyboard and mouse sharing service also needs to hide the cursor 10a displayed on the main screen, and display the cursor 10b on the pad screen, specifically at the mouse position determined by the Sink keyboard and mouse sharing service.
[0194] Continue to see Figure 6 , for example, when the mouse passes to Figure 7B After the gray area of the Android device screen (pad screen) is shown in (2), if the user continues to press the mouse coordinates to move the mouse, during the mouse movement, the Source keyboard and mouse sharing service will obtain the current mouse offset and send the mouse offset to the Sink keyboard and mouse sharing service through the Source data transmission unit and the Sink data transmission unit.
[0195] Continue to see Figure 6 For example, the Sink keyboard and mouse sharing service will recalculate the mouse position after the offset based on the received mouse offset and the previously determined mouse coordinates, and determine whether the mouse has moved into the extended screen area, is still outside the extended screen area, or has moved to the edge of the pad (traveling back to the edge of the main screen of the Windows device) based on the recalculated mouse position.
[0196] For the sake of convenience, this embodiment takes the case where the mouse moves into the extended screen area as an example. Figure 6 For example, when it is determined that the mouse is located in the extended screen area displayed on the pad, the Sink keyboard and mouse sharing service will inform the Source keyboard and mouse sharing service through the Sink data transmission unit and the Source data transmission unit that the mouse in the current scene has crossed into the extended screen, that is, Figure 6 The confirmation shown in the figure shows that the mouse has crossed into the extended screen, such as from Figure 7C (1) becomes Figure 7C Chinese (2).
[0197] From the above description of the cursor styles on the main screen, extended screen and pad screen, it can be seen that when the mouse crosses to the main screen or extended screen, the corresponding cursor style of the mouse is cursor 10a. Therefore, when the mouse crosses the extended screen, the corresponding cursor of the mouse will change from Figure 7C (1) The cursor 10b displayed on the Android screen (gray area) switches to Figure 7C(2) The cursor 10a is displayed on the extended screen.
[0198] pass Figure 7C Zhong (1) and Figure 7C As can be seen from the switching of the mouse shown in (2), when the mouse passes from the pad screen to the extended screen, the Source keyboard and mouse sharing service also needs to hide the cursor 10b displayed on the gray area of the pad screen, and display the cursor 10a on the extended screen, specifically at the mouse position determined by the Sink keyboard and mouse sharing service.
[0199] It should be noted that in a possible implementation, for the scenario where the mouse passes from the gray area of the pad screen to the extended screen, since the cursor moves with the movement of the mouse, the cursor 10a that moves from the pad screen to the extended screen can be directly replaced with cursor 10a.
[0200] The above description is merely an example listed for a better understanding of the technical solution of this embodiment and is not intended to be the sole limitation to this embodiment.
[0201] In addition, it should be noted that, since the operation performed in the extended screen after the mouse crosses to the extended screen is implemented by the screen collaborative service, the operation of setting the cursor 10a to the extended screen area when the mouse crosses to the extended screen needs to be completed by the screen collaborative service.
[0202] Continue to see Figure 6 For example, if the mouse moves to the extended screen and the user still presses the left button of the mouse to move it, when the mouse moves in the extended screen, the Source keyboard and mouse sharing service senses the movement of the mouse and starts the extended screen edge detection process. For example, the coordinate information of the cursor 10a on the extended screen is obtained, and then the coordinate information of the cursor 10a is used to determine whether the cursor 10a has moved to the edge of the extended screen. Figure 7C In (1), the area with the X-axis coordinate x1, or the area with the Y-axis coordinate y1, or the area with the Y-axis coordinate y2 is determined. That is, it is determined whether the cursor 10a moves to the upper edge, the lower edge, the left edge, or the right edge of the extended screen, etc. Figure 7C (2) shows the four black line areas of the extended screen.
[0203] Continue to see Figure 6 For example, when the coordinate information of the mouse is outside the extended screen area, such as returning to the gray area of the pad screen, it is determined that the mouse exceeds the edge of the extended screen. At this time, the mouse will cross, that is, from the extended screen to the gray area of the pad screen.
[0204] Continue to see Figure 6For example, when it is determined that the mouse exceeds the edge of the extended screen, the Source keyboard and mouse sharing service will initiate a mouse crossing request to the Sink keyboard and mouse sharing service. Figure 6 The mouse crossing request will be sent to the Source data transmission unit through the Source keyboard and mouse sharing service, and then sent to the Sink data transmission unit by the Source data transmission unit based on the MagicLink transmission technology, and finally sent to the Sink keyboard and mouse sharing service by the Sink data transmission unit.
[0205] Continue to see Figure 6 For example, after the mouse crossing request reaches the Sink keyboard and mouse sharing service, the Sink keyboard and mouse sharing service will redetermine the current mouse position of the mouse, and based on the redetermined mouse position, determine whether the mouse has moved outside the extended screen area, is still within the extended screen area, or has directly crossed back to the main screen.
[0206] For the sake of convenience, this embodiment takes the mouse being outside the extended screen area as an example. Figure 6 For example, when it is determined that the mouse is in the gray area of the pad screen, the Sink keyboard and mouse sharing service will inform the Source keyboard and mouse sharing service through the Sink data transmission unit and the Source data transmission unit that the mouse in the current scene has crossed to the pad screen, that is, Figure 6 The confirmation mouse crosses to the pad screen as shown in Figure 7C (2) becomes Figure 7C Chinese (1).
[0207] From the above description of the cursor styles on the main screen, extended screen, and pad screen, we can see that when the mouse crosses to the pad screen, the cursor corresponding to the mouse will change from Figure 7C (2) The cursor 10a displayed on the extended screen switches to Figure 7C (1) Cursor 10b displayed on the pad screen.
[0208] pass Figure 7C Zhong (2) and Figure 7C As can be seen from the switching of the mouse shown in (1), when the mouse passes from the extended screen to the pad screen, the Source keyboard and mouse sharing service also needs to hide the cursor 10a displayed on the extended screen, and display the cursor 10b on the pad screen, specifically at the mouse position determined by the Sink keyboard and mouse sharing service.
[0209] In addition, it should be noted that, considering the actual application, fixed taskbars may be displayed or set around the extended screen, such as the upper edge, lower edge, left edge, and right edge. The operation of these fixed taskbars is based on the keyboard and mouse sharing function, not the screen collaboration function. Therefore, in some possible implementations, it can be set that when the mouse moves from the extended screen area to the upper edge, lower edge, left edge, or right edge of the extended screen, it is confirmed that the mouse has crossed, that is, from the extended screen to the pad screen.
[0210] Accordingly, when crossing from the pad screen to the extended screen, the mouse can be confirmed to have crossed only when it moves into the extended screen from the upper edge, lower edge, left edge, or right edge of the extended screen, and that it has crossed from the pad screen to the extended screen.
[0211] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation to this embodiment.
[0212] Therefore, the method for concurrent screen collaboration and keyboard and mouse sharing provided in the embodiment of the present application solves the problem that the existing screen collaboration connection and keyboard and mouse sharing connection conflict and cannot be concurrent. At the same time, based on the above mouse switching process, users can control electronic devices that establish screen collaboration connection based on the same set of keyboard and mouse.
[0213] In addition, it is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.
[0214] In addition, it should be noted that the concurrent methods of screen collaboration and keyboard and mouse sharing provided by the above embodiments implemented by electronic devices in actual application scenarios can also be executed by a chip system included in the electronic device, wherein the chip system may include a processor. The chip system can be coupled to a memory so that the chip system calls a computer program stored in the memory when it is running to implement the steps executed by the above electronic device. The processor in the chip system can be an application processor or a processor other than an application processor.
[0215] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the concurrent method of screen collaboration and keyboard and mouse sharing in the above-mentioned embodiment.
[0216] In addition, an embodiment of the present application also provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the above-mentioned related steps to implement the concurrent method of screen collaboration and keyboard and mouse sharing in the above-mentioned embodiment.
[0217] In addition, an embodiment of the present application also provides a chip (which may also be a component or module), which may include one or more processing circuits and one or more transceiver pins; wherein the transceiver pins and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the above-mentioned related method steps to implement the concurrent method of screen collaboration and keyboard and mouse sharing in the above-mentioned embodiment, so as to control the receiving pin to receive signals, so as to control the sending pin to send signals.
[0218] In addition, it can be seen from the above description that the electronic device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.
[0219] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A concurrent method for screen collaboration and keyboard and mouse sharing, It is characterized in that Applied to a system including a first electronic device and a second electronic device, the method includes: In response to the operation of establishing the screen cooperative connection, the first electronic device searches for the second electronic device based on the first connection technology; When the second electronic device is searched based on the first connection technology, the first electronic device determines whether the second electronic device can be searched through the second connection technology; When the second electronic device is searched based on the second connection technology, the first electronic device establishes a screen cooperative connection with the second electronic device based on the second connection technology; wherein, based on the second connection technology, the first electronic device can also establish a keyboard and mouse sharing connection with the second electronic device, and the second connection technology is used to simultaneously establish the screen cooperative connection and the keyboard and mouse sharing connection between the first electronic device and the second electronic device based on the same data transmission unit; When the second electronic device is not found based on the second connection technology, the first electronic device establishes a screen cooperative connection with the second electronic device based on the first connection technology; The first connection technology and the second connection technology are two mutually exclusive connection technologies, and at the same time, the first electronic device can only establish a communication connection with the second electronic device based on the first connection technology or the second connection technology; When the first electronic device establishes a communication connection with the second electronic device based on the first connection technology, the communication connection is the screen cooperative connection; When the first electronic device establishes a communication connection with the second electronic device based on the second connection technology, the communication connection is the screen cooperative connection and the keyboard and mouse sharing connection.
2. The method according to claim 1, It is characterized in that When the second electronic device is searched based on the first connection technology, the method further includes: Determining, by the first electronic device, whether the second electronic device supports establishing the screen cooperative connection by using the second connection technology; When it is determined that the second electronic device supports establishing the screen cooperative connection using the second connection technology, the first electronic device executes the step of establishing the screen cooperative connection with the second electronic device based on the second connection technology; When it is determined that the second electronic device does not support establishing the screen cooperative connection using the second connection technology, the first electronic device executes the step of establishing the screen cooperative connection with the second electronic device based on the first connection technology.
3. The method according to claim 2, It is characterized in that The first electronic device determines whether the second electronic device supports establishing the screen cooperative connection by using the second connection technology, including: The first electronic device obtains a version number supported by the second electronic device; The first electronic device determines whether the version number supported by the second electronic device is the same as the version number supported by the first electronic device; wherein the version number supported by the first electronic device and the version number supported by the second electronic device are both version numbers of functional modules implemented based on the second connection technology; When it is determined that the version number supported by the second electronic device is the same as the version number supported by the first electronic device, determining that the second electronic device supports establishing the screen collaborative connection using the second connection technology; When it is determined that the version number supported by the second electronic device is different from the version number supported by the first electronic device, it is determined that the second electronic device does not support establishing the screen cooperative connection by using the second connection technology.
4. The method according to claim 2, It is characterized in that The first electronic device determines whether the second electronic device supports establishing the screen cooperative connection by using the second connection technology, including: The first electronic device obtains a version number supported by the second electronic device; The first electronic device determines whether the version number supported by the second electronic device is the same as a preset version number; wherein the version number supported by the second electronic device and the preset version number are both version numbers of a functional module implemented based on the second connection technology; When it is determined that the version number supported by the second electronic device is the same as the preset version number, determining that the second electronic device supports establishing the screen collaborative connection using the second connection technology; When it is determined that the version number supported by the second electronic device is different from the preset version number, it is determined that the second electronic device does not support establishing the screen cooperative connection by using the second connection technology.
5. The method according to any one of claims 2 to 4, It is characterized in that The version number supported by the second electronic device is written into the functional module integrated with the second connection technology by the screen cooperation service of the second electronic device when the second electronic device is powered on.
6. The method according to claim 1, It is characterized in that When the second electronic device is searched based on the second connection technology, after the first electronic device establishes a screen cooperative connection with the second electronic device based on the second connection technology, the method further includes: Based on the second connection technology, the first electronic device establishes a keyboard and mouse sharing connection with the second electronic device.
7. The method according to claim 6, It is characterized in that In a scenario where screen collaboration and keyboard and mouse sharing are performed concurrently, and a floating window serving as an extended screen of the first electronic device is displayed on the screen of the second electronic device, the method further includes: When the first electronic device detects that the mouse moves within the main screen of the first electronic device, main screen edge detection is started; wherein the mouse is connected to the first electronic device; When it is determined through the main screen edge detection that the mouse moves out of the main screen, the first electronic device sends a first mouse crossing request to the second electronic device, where the first mouse crossing request carries a first mouse offset; After receiving the first mouse crossing request, the second electronic device calculates the first mouse position according to the first mouse offset; When the first mouse position is outside the extended screen, the second electronic device makes a first confirmation message for the first mouse crossing request and sends the first confirmation message to the first electronic device; wherein the first confirmation message indicates that the mouse crosses from the main screen to the screen of the second electronic device; The first electronic device hides the first style mouse cursor displayed on the main screen, and based on the keyboard and mouse sharing function, displays a second style mouse cursor on the screen of the second electronic device.
8. The method according to claim 7, It is characterized in that The method further comprises: When the first mouse position is within the extended screen, the second electronic device makes a second confirmation message for the first mouse crossing request, and sends the second confirmation message to the first electronic device; wherein the second confirmation message indicates that the mouse crosses from the main screen to the extended screen; The first electronic device hides the mouse cursor of the first style displayed on the main screen, and displays the mouse cursor of the first style on the extended screen based on the screen cooperative function.
9. The method according to claim 7, It is characterized in that The method further comprises: During the process of displaying the second-style mouse cursor on the screen of the second electronic device, when the mouse continues to be moved, the first electronic device acquires a second mouse offset of the mouse, and sends the second mouse offset to the second electronic device; The second electronic device calculates a second mouse position according to the second mouse offset; When the second mouse position is within the extended screen, the second electronic device generates a third confirmation message; wherein the third confirmation message indicates that the mouse has passed from the screen of the second electronic device to the extended screen; The first electronic device replaces the second-style mouse cursor with the first-style mouse cursor, and sets the first-style mouse cursor in the extended screen based on a screen collaboration function.
10. The method according to claim 8 or 9, It is characterized in that The method further comprises: During the process of displaying the mouse cursor of the first style in the extended screen, when the mouse continues to be moved, the first electronic device starts extended screen edge detection; When it is determined through the extended screen edge detection that the mouse moves out of the extended screen, the first electronic device sends a second mouse crossing request to the second electronic device, wherein the second mouse crossing request carries a third mouse offset; After receiving the second mouse crossing request, the second electronic device calculates the third mouse position according to the third mouse offset; When the third mouse position is outside the extended screen, the second electronic device makes a fourth confirmation message for the second mouse crossing request, and sends the fourth confirmation message to the first electronic device; wherein the fourth confirmation message indicates that the mouse crosses from the extended screen to the screen of the second electronic device; The first electronic device hides the first style mouse cursor displayed on the extended screen, and based on the keyboard and mouse sharing function, displays a second style mouse cursor on the screen of the second electronic device.
11. An electronic device, It is characterized in that The electronic device includes: a memory and a processor, the memory and the processor are coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the concurrent method of screen collaboration and keyboard and mouse sharing as described in any one of claims 1 to 10.
12. A computer-readable storage medium, It is characterized in that It comprises a computer program, which, when executed on an electronic device, enables the electronic device to execute the concurrent method of screen collaboration and keyboard and mouse sharing as claimed in any one of claims 1 to 10.
Citation Information
Patent Citations
Interface display method and device
CN114115629A
Method for lightening screen and electronic equipment
CN114816153A
Screen sharing method and system, electronic equipment and Android equipment
CN115695383A
Cross-device object moving method, electronic device and system
CN116360657A
Method for sharing keyboard and mouse among multiple devices and electronic device
CN116400814A
