Multi-device coordination method and related apparatus

Abstract

Disclosed in the present application are a multi-device coordination method and a related apparatus. The method comprises: a first terminal establishes a communication connection with a second terminal, the second terminal being connected to a first input device; the first terminal displays a first user interface; the second terminal shares at least one input device with the first terminal, the first terminal and the second terminal being in an input device sharing mode; in response to a first operation of the first input device, the first terminal displays that a cursor of the first input device drags a first interface snapshot of the first user interface; and, after the cursor has dragged the first interface snapshot to a display screen of the second terminal, the second terminal and the first terminal switch the input device sharing mode to a multi-screen coordination mode, and the second terminal displays a multi-screen coordination window corresponding to the first terminal, the multi-screen coordination window displaying content associated with the first user interface. Thus, the present application can achieve coordination modes matched with real-time demands of users without complex operations, thereby effectively improving user experiences.

Description

Multi-device collaboration method and related device

[0001] This application claims priority to Chinese patent application No. 202311660574.7 filed with the State Intellectual Property Office of China on December 5, 2023, and priority to Chinese patent application No. 202311660574.7, entitled “Multi-device collaboration method and related apparatus”, the entire contents of which are incorporated herein by reference. This application claims priority to Chinese patent application No. 202311873903.6 filed with the State Intellectual Property Office of China on December 30, 2023, and priority to Chinese patent application No. 202311873903.6, entitled “Multi-device collaboration method and related apparatus”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of electronic technology, and in particular to a multi-device collaboration method and related apparatus. Background Art

[0003] With the continuous development of terminal technology, users have more and more types of electronic devices with more functions. Due to the limitations of their own hardware structure and system performance, different types of devices usually have different capabilities and advantages. The user's multiple devices, combined with the performance advantages of each device, conduct collaborative control of device interconnection and distributed services, bringing a better user experience to users. After multiple devices are collaboratively connected, multiple devices can collaborate to achieve multiple functional modes. For example, mobile terminals such as mobile phones can migrate multimedia content (such as pictures, audio and video, etc.) to smart screens for continued playback and projection; laptops can share the mouse with tablets, and users can control the tablets through the laptop mouse, and so on.

[0004] After establishing a communication connection between devices, it remains to be studied what functional mode to choose for collaborative implementation to match the user's real-time needs as much as possible. Summary of the Invention

[0005] The embodiment of the present application provides a multi-device collaboration method that can appropriately select a collaboration mode to match the user's real-time needs as much as possible without the need for complex operations, thereby effectively improving the user experience.

[0006] In the first aspect, the present application provides a multi-device collaboration method, which is applied to a multi-device collaboration system, wherein the multi-device collaboration system includes a first terminal and a second terminal, and the second terminal is connected to a first input device. The method includes: the first terminal establishes a communication connection with the second terminal; the first terminal displays a first user interface; the second terminal shares at least one input device with the first terminal, and the first terminal and the second terminal are in an input device sharing mode, and the above-mentioned at least one input device includes a first input device; in response to a first operation of the first input device, the first terminal displays a first interface snapshot of the first user interface dragged by the cursor of the first input device; after the cursor drags the first interface snapshot to the display screen of the second terminal, the second terminal and the first terminal switch the input device sharing mode to a multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal, and the multi-screen collaboration window displays content associated with the first user interface.

[0007] In the implementation of the embodiment of the present application, in the input device sharing mode, dragging the user interface displayed on the first terminal to the screen of the second terminal through the input device of the second terminal can trigger the first terminal and the second terminal to switch the input device sharing mode to the multi-screen collaborative mode. The multi-screen collaborative window displayed by the second terminal can be used to control the first terminal without the need to control the first terminal through the input device of the second terminal. In this way, according to the user's drag operation, the multi-screen collaborative mode can be quickly selected and switched to, which matches the user's real-time demand for the collaborative mode between devices and effectively improves the user experience.

[0008] In one implementation, the first user interface is the user interface of the first application, and the multi-screen collaboration window displays the user interface of the first application. The method further includes: the second terminal detecting a second operation acting on the multi-screen collaboration window; and in response to the second operation, the second terminal displaying the user interface of the second application in the multi-screen collaboration window. In implementing this embodiment of the present application, in the multi-screen collaboration mode, a user can manipulate the multi-screen collaboration window of the second terminal to display the user interfaces of different applications of the first terminal.

[0009] In one implementation, the first user interface is a system interface. When implementing the embodiment of the present application, in the multi-screen collaboration mode, the multi-screen collaboration window of the second terminal can display the system interface of the first terminal.

[0010] In one implementation, the distance between the cursor and the boundary of the first interface snapshot is a first value, and the method further includes: when the cursor drags the first interface snapshot to move on the screen of the first terminal, when the distance between the cursor and the boundary of the first terminal's screen is less than the first value and continues to decrease, the first area of ​​the first interface snapshot moves out of the screen of the first terminal and into the screen of the second terminal. In implementing this embodiment of the present application, when the cursor drags the interface snapshot to move, a portion of the interface snapshot moves out of the screen of the first terminal and into the screen of the second terminal. In this way, the visual effect is smoother and more beautiful.

[0011] In one implementation, after the cursor drags a first interface snapshot to the screen of a second terminal, the first terminal displays the first area of ​​the first interface snapshot, and the second terminal displays a second area of ​​the first interface snapshot excluding the first area. Alternatively, after the cursor drags the first interface snapshot to the screen of the second terminal, the first terminal does not display the first interface snapshot, and the second terminal displays a portion of the first interface snapshot. In implementing an embodiment of the present application, when the cursor is dragging the interface snapshot to move, when the cursor moves to the second terminal, while the second terminal displays a portion of the interface snapshot, the first terminal can display the remaining area of ​​the interface snapshot or stop displaying the interface snapshot, thereby enriching the visual effect of dragging the interface snapshot.

[0012] In one implementation, when the first user interface belongs to an application that cannot be applied and continued by the second terminal, the multi-screen collaboration mode is a projection mode; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays the first user interface; when the first user interface belongs to the first application that can be applied and continued by the second terminal, the multi-screen collaboration mode is an application continuation mode; the first user interface displayed by the first terminal is the user interface of the first version of the first application; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays the user interface corresponding to the first user interface in the second version of the first application; the first version is an application version applicable to the device type of the first terminal, and the second version is an application version applicable to the device type of the second terminal. When implementing the embodiment of the present application, without user selection, it is possible to adaptively determine whether the multi-screen collaboration mode is specifically the projection mode or the application continuation mode based on the page running in the foreground.

[0013] In one implementation, the multi-screen collaboration mode is a projection mode; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays the first user interface. When switching to the projection mode, the user interface currently displayed on the first terminal can be projected to the multi-screen collaboration window of the second terminal.

[0014] In one implementation, the multi-screen collaboration mode is an application continuation mode; the first user interface displayed by the first terminal is the user interface of the first version of the first application; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays a user interface corresponding to the first user interface in the second version of the first application; the first version is an application version applicable to the device type of the first terminal, and the second version is an application version applicable to the device type of the second terminal. When implementing an embodiment of the present application, the first terminal runs the user interface of the first application in the foreground, and when switching to the application continuation mode, the second terminal opens the first application installed on this device in the multi-screen collaboration window, and then continues to display the first application.

[0015] In one implementation, the first user interface belongs to a first application that can be applied to the continuation mode. When the second terminal has not installed the first application, before switching the input device sharing mode to the multi-screen collaboration mode, the method includes: the second terminal displays a prompt message, the prompt message is used to prompt the user to install the first application on the second terminal; in response to a third operation, the second terminal installs the first application and determines that the multi-screen collaboration mode is the application continuation mode; in response to a fourth operation, the multi-screen collaboration mode is determined to be the projection mode. In implementing the embodiment of the present application, when the first terminal runs the first application in the foreground and switches to the application continuation mode, if the second terminal has not installed the first application, it can remind the user to install the first application and switch to the application continuation mode after the installation is successful, otherwise it switches to the projection mode, thereby improving the user experience.

[0016] In one implementation, the method further includes: when the first terminal and the second terminal are in multi-screen collaborative mode, in response to the fifth operation of the first input device, the second terminal displays the cursor of the first input device and drags the multi-screen collaborative window to move; during the process of dragging the multi-screen collaborative window, after the cursor moves to the boundary of the screen of the second terminal, the second terminal and the first terminal switch the multi-screen collaborative mode to the input device sharing mode, the second terminal stops displaying the multi-screen collaborative window, and shares the above-mentioned at least one input device with the first terminal. In implementing the embodiment of the present application, in the multi-screen collaborative mode, the second terminal drags the multi-screen collaborative window to the screen boundary, which can trigger the multi-screen collaborative mode to switch to the input device sharing mode. Then, the input device of the second terminal can be used to directly control the first terminal, and there is no need to control the first terminal through the multi-screen collaborative window. In this way, according to the user's drag operation, quick selection and switching to the input device sharing mode are achieved, which matches the user's real-time demand for the collaborative mode between devices and effectively improves the user experience.

[0017] In one implementation, the multi-screen collaboration window displays a second user interface; when the multi-screen collaboration mode is switched to the input device sharing mode, the method further includes: the first terminal displays content associated with the second user interface.

[0018] In one implementation, the multi-screen collaboration mode is a projection mode, and the first terminal displays content associated with the second user interface, including: the first terminal displays the second user interface. In this way, when the projection mode is switched to the input device sharing mode, the first terminal can also continue to display the projected content in the multi-screen collaboration window.

[0019] In one implementation, the multi-screen collaboration mode is an application continuation mode, the second user interface displayed in the multi-screen collaboration window is the user interface of the second version of the first application, and the first terminal displays content associated with the second user interface, including: the first terminal displays the user interface corresponding to the second user interface in the first version of the first application; the first version is the application version applicable to the device type of the first terminal; and the second version is the application version applicable to the device type of the second terminal. In this way, when the application continuation mode is switched to the input device sharing mode, the first terminal can also continue to display the application in the multi-screen collaboration window.

[0020] In one implementation, the method further includes: when the first terminal and the second terminal are not in input device sharing mode and multi-screen collaboration mode, the first terminal or the second terminal receives and responds to the sixth operation, and when the device status of the first terminal meets the first start-up condition, the first terminal and the second terminal are triggered to start the input device sharing mode, and the second terminal shares the above-mentioned at least one input device with the first terminal; when the device status of the first terminal meets the second start-up condition, the first terminal and the second terminal are triggered to start the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal. When implementing the embodiment of the present application, when starting the collaboration mode, the collaboration mode that matches the device status of the first terminal is automatically selected from multiple collaboration modes to meet the real-time needs of the user as much as possible.

[0021] In one implementation, the method further includes: when the first terminal and the second terminal are in multi-screen collaborative mode, upon detecting that the device status of the first terminal satisfies a first starting condition, triggering the first terminal and the second terminal to switch the multi-screen collaborative mode to an input device sharing mode; and when the first terminal and the second terminal are in input device sharing mode, upon detecting that the device status of the first terminal satisfies a second starting condition, triggering the first terminal and the second terminal to switch the input device sharing mode to the multi-screen collaborative mode. By implementing this embodiment of the present application, the device status of the first terminal can be detected, and the collaborative mode can be automatically switched to a collaborative mode that matches the device status.

[0022] In one implementation, the first terminal and the second terminal switch the input device sharing mode to the multi-screen collaboration mode, including: when the second terminal cannot apply the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the screen projection mode, and the second terminal displays the projected content from the first terminal in the multi-screen collaboration window; when the second terminal can apply the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the application continuation mode, and the second terminal performs the application continuation on the application running in the foreground of the first terminal in the multi-screen collaboration window.

[0023] In one implementation, the first startup condition includes one or more of the following: not in the anti-mistouch scenario, the anti-mistouch function is not turned on, the screen is facing the user, the posture is inclined to be upright, and the distance from the second terminal is less than a third value; the second startup condition includes: the device status of the first terminal does not meet the first startup condition.

[0024] In the second aspect, the present application provides a multi-device collaboration method, characterized in that the method includes: a first terminal establishes a communication connection with a second terminal, and the second terminal is connected to a first input device; the first terminal displays a first user interface; the first terminal and the second terminal are in an input device sharing mode, and the first terminal shares at least one input device of the second terminal, and the above-mentioned at least one input device includes a first input device; in response to a first operation of the first input device, the first terminal displays a first interface snapshot of the first user interface dragged by the cursor of the first input device; after the cursor drags the first interface snapshot out of the screen of the first terminal, the first terminal switches the input device sharing mode with the second terminal to a multi-screen collaboration mode, and sends collaboration data of the multi-screen collaboration window to the second terminal, the collaboration data is used to determine the display content of the multi-screen collaboration window, and the multi-screen collaboration window is used to be displayed on the screen of the second terminal. The multi-screen collaboration window is used to display content associated with the first user interface.

[0025] In the implementation of the embodiment of the present application, in the input device sharing mode, dragging the user interface displayed on the first terminal to the screen of the second terminal through the input device of the second terminal can trigger the first terminal and the second terminal to switch the input device sharing mode to the multi-screen collaborative mode. The multi-screen collaborative window displayed by the second terminal can be used to control the first terminal without the need to control the first terminal through the input device of the second terminal. In this way, according to the user's drag operation, the multi-screen collaborative mode can be quickly selected and switched to, which matches the user's real-time demand for the collaborative mode between devices and effectively improves the user experience.

[0026] In one implementation, the first user interface is the user interface of the first application, the multi-screen collaboration window is used to display content in the first application that is associated with the first user interface, and the multi-screen collaboration window is also used to display the user interface of the second application. In implementing the embodiment of the present application, in the multi-screen collaboration mode, the user can manipulate the multi-screen collaboration window of the second terminal to display the user interfaces of different applications of the first terminal.

[0027] In one implementation, the first user interface is a system interface. When implementing the embodiment of the present application, in the multi-screen collaboration mode, the multi-screen collaboration window of the second terminal can display the system interface of the first terminal.

[0028] In one implementation, when the first user interface belongs to an application that cannot be continued by the second terminal, the multi-screen collaboration mode is the projection mode; the content associated with the first user interface is the first user interface; when the first user interface belongs to the first application that can be continued by the second terminal, the multi-screen collaboration mode is the application continuation mode; the first user interface displayed by the first terminal is the user interface of the first version of the first application; the content associated with the first user interface includes the user interface corresponding to the first user interface in the second version of the first application; the first version is an application version applicable to the device type of the first terminal, and the second version is an application version applicable to the device type of the second terminal. When implementing the embodiment of the present application, the multi-screen collaboration mode can be adaptively determined to be the projection mode or the application continuation mode based on the page running in the foreground without the need for user selection.

[0029] In one implementation, the method further includes: when the first terminal and the second terminal are in a multi-screen collaborative mode, in response to a first message sent by the second terminal, the first terminal switches the multi-screen collaborative mode with the second terminal to an input device sharing mode; the first message is sent after the cursor drags the multi-screen collaborative window on the screen of the second terminal and the cursor moves to the boundary of the screen of the second terminal. In implementing the embodiment of the present application, in the multi-screen collaborative mode, the second terminal drags the multi-screen collaborative window to the screen boundary, which can trigger the multi-screen collaborative mode to be switched to the input device sharing mode. Then, the input device of the second terminal can be used to directly control the first terminal, and there is no need to control the first terminal through the multi-screen collaborative window. In this way, according to the user's dragging operation, the input device sharing mode is quickly selected and switched to, which matches the user's real-time demand for the collaborative mode between devices, and effectively improves the user experience.

[0030] In one implementation, the multi-screen collaboration window displays a second user interface; when the multi-screen collaboration mode with the second terminal is switched to an input device sharing mode, the method further includes: the first terminal displays content associated with the second user interface.

[0031] In one implementation, the multi-screen collaboration mode is a projection mode, and the first terminal displays content associated with the second user interface, including: the first terminal displays the second user interface. In this way, when the projection mode is switched to the input device sharing mode, the first terminal can also continue to display the projected content in the multi-screen collaboration window.

[0032] In one implementation, the multi-screen collaboration mode is an application continuation mode, the second user interface displayed in the multi-screen collaboration window is the user interface of the second version of the first application, and the first terminal displays content associated with the second user interface, including: the first terminal displays the user interface corresponding to the second user interface in the first version of the first application; the first version is the application version applicable to the device type of the first terminal; and the second version is the application version applicable to the device type of the second terminal. In this way, when the application continuation mode is switched to the input device sharing mode, the first terminal can also continue to display the application in the multi-screen collaboration window.

[0033] In the third aspect, the present application provides a multi-device collaboration method, which includes: the second terminal establishes a communication connection with the first terminal, and the second terminal is connected to a first input device; the second terminal shares the above-mentioned at least one input device with the first terminal, and the first terminal and the second terminal are in input device sharing mode, and the above-mentioned at least one input device includes the first input device; after the cursor of the first input device drags the first interface snapshot of the first user interface displayed on the first terminal to the display screen of the second terminal, the second terminal switches the input device sharing mode with the first terminal to the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal, and the multi-screen collaboration window displays content associated with the first user interface.

[0034] In the implementation of the embodiment of the present application, in the input device sharing mode, dragging the user interface displayed on the first terminal to the screen of the second terminal through the input device of the second terminal can trigger the first terminal and the second terminal to switch the input device sharing mode to the multi-screen collaborative mode. The multi-screen collaborative window displayed by the second terminal can be used to control the first terminal without the need to control the first terminal through the input device of the second terminal. In this way, according to the user's drag operation, the multi-screen collaborative mode can be quickly selected and switched to, which matches the user's real-time demand for the collaborative mode between devices and effectively improves the user experience.

[0035] In one implementation, the first user interface is the user interface of the first application, and the multi-screen collaboration window displays the user interface of the first application. The method further includes: the second terminal detecting a second operation acting on the multi-screen collaboration window; and in response to the second operation, the second terminal displaying the user interface of the second application in the multi-screen collaboration window. In implementing this embodiment of the present application, in the multi-screen collaboration mode, a user can manipulate the multi-screen collaboration window of the second terminal to display the user interfaces of different applications of the first terminal.

[0036] In one implementation, the first user interface is a system interface. When implementing the embodiment of the present application, in the multi-screen collaboration mode, the multi-screen collaboration window of the second terminal can display the system interface of the first terminal.

[0037] In one implementation, when the first user interface belongs to an application that cannot be applied and continued by the second terminal, the multi-screen collaboration mode is a projection mode; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays the first user interface; when the first user interface belongs to the first application that can be applied and continued by the second terminal, the multi-screen collaboration mode is an application continuation mode; the first user interface displayed by the first terminal is the user interface of the first version of the first application; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays the user interface corresponding to the first user interface in the second version of the first application; the first version is an application version applicable to the device type of the first terminal, and the second version is an application version applicable to the device type of the second terminal. When implementing the embodiment of the present application, without user selection, it is possible to adaptively determine whether the multi-screen collaboration mode is specifically the projection mode or the application continuation mode based on the page running in the foreground.

[0038] In one implementation, the first user interface belongs to a first application that can be applied to the continuation mode. When the second terminal has not installed the first application, before switching the input device sharing mode to the multi-screen collaboration mode, the method includes: the second terminal displays a prompt message, the prompt message is used to prompt the user to install the first application on the second terminal; in response to a third operation, the second terminal installs the first application and determines that the multi-screen collaboration mode is the application continuation mode; in response to a fourth operation, the multi-screen collaboration mode is determined to be the projection mode. In implementing the embodiment of the present application, when the first terminal runs the first application in the foreground and switches to the application continuation mode, if the second terminal has not installed the first application, it can remind the user to install the first application and switch to the application continuation mode after the installation is successful, otherwise it switches to the projection mode, thereby improving the user experience.

[0039] In one implementation, the method further includes: when the first terminal and the second terminal are in multi-screen collaborative mode, in response to the fifth operation of the first input device, the second terminal displays the cursor of the first input device and drags the multi-screen collaborative window to move; during the process of dragging the multi-screen collaborative window, after the cursor moves to the boundary of the screen of the second terminal, the second terminal and the first terminal switch the multi-screen collaborative mode to the input device sharing mode, the second terminal stops displaying the multi-screen collaborative window, and shares the above-mentioned at least one input device with the first terminal. In implementing the embodiment of the present application, in the multi-screen collaborative mode, the second terminal drags the multi-screen collaborative window to the screen boundary, which can trigger the multi-screen collaborative mode to switch to the input device sharing mode. Then, the input device of the second terminal can be used to directly control the first terminal, and there is no need to control the first terminal through the multi-screen collaborative window. In this way, according to the user's drag operation, quick selection and switching to the input device sharing mode are achieved, which matches the user's real-time demand for the collaborative mode between devices and effectively improves the user experience.

[0040] In a fourth aspect, the present application provides a multi-device collaboration method, which is applied to a multi-device collaboration system, wherein the multi-device collaboration system includes a first terminal and a second terminal, and the method includes: when the first terminal and the second terminal are not in input device sharing mode and multi-screen collaboration mode, the first terminal or the second terminal receives and responds to a sixth operation, and when the device status of the first terminal meets the first start-up condition, the first terminal and the second terminal are triggered to start the input device sharing mode, and the second terminal shares the above-mentioned at least one input device with the first terminal; when the device status of the first terminal meets the second start-up condition, the first terminal and the second terminal are triggered to start the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal. When implementing an embodiment of the present application, when starting the collaboration mode, the collaboration mode that matches the device status of the first terminal is automatically selected from multiple collaboration modes to meet the real-time needs of the user as much as possible.

[0041] In one implementation, the method further includes: when the first terminal and the second terminal are in multi-screen collaborative mode, upon detecting that the device status of the first terminal satisfies a first starting condition, triggering the first terminal and the second terminal to switch the multi-screen collaborative mode to an input device sharing mode; and when the first terminal and the second terminal are in input device sharing mode, upon detecting that the device status of the first terminal satisfies a second starting condition, triggering the first terminal and the second terminal to switch the input device sharing mode to the multi-screen collaborative mode. By implementing this embodiment of the present application, the device status of the first terminal can be detected, and the collaborative mode can be automatically switched to a collaborative mode that matches the device status.

[0042] In one implementation, the first terminal and the second terminal switch the input device sharing mode to the multi-screen collaboration mode, including: when the second terminal cannot apply the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the screen projection mode, and the second terminal displays the projected content from the first terminal in the multi-screen collaboration window; when the second terminal can apply the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the application continuation mode, and the second terminal performs the application continuation on the application running in the foreground of the first terminal in the multi-screen collaboration window.

[0043] In one implementation, the first startup condition includes one or more of the following: not in the anti-mistouch scenario, the anti-mistouch function is not turned on, the screen is facing the user, the posture is inclined to be upright, and the distance from the second terminal is less than a third value; the second startup condition includes: the device status of the first terminal does not meet the first startup condition.

[0044] In the fifth aspect, an embodiment of the present application provides a terminal device, which includes: a processor and a memory, the memory being coupled to the processor, the memory being used to store computer program code, the computer program code including computer instructions, and when the processor reads the computer instructions from the memory, the terminal device executes the multi-device collaboration method described in the first aspect.

[0045] In a sixth aspect, an embodiment of the present application provides a computer storage medium comprising computer instructions, which, when executed on a terminal device, enables the terminal device to execute a multi-device collaboration method in any possible implementation of any of the above aspects.

[0046] In a seventh aspect, an embodiment of the present application provides a computer program product, which, when running on a computer, enables the computer to execute the multi-device collaboration method in any possible implementation of any of the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] FIG1 is a schematic diagram of a system architecture of a communication system provided in an embodiment of the present application;

[0048] FIG2A is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0049] FIG2B is a schematic diagram of a geographic coordinate system provided in an embodiment of the present application;

[0050] FIG2C is a schematic diagram of a device coordinate system provided in an embodiment of the present application;

[0051] FIG3 is a schematic diagram of a software system architecture provided in an embodiment of the present application;

[0052] 4A to 4I are schematic diagrams of interfaces for automatically selecting a collaborative mode according to an embodiment of the present application;

[0053] 5A to 5B are schematic diagrams of the interface of the hyperterminal page of a mobile phone provided in an embodiment of the present application;

[0054] 6A to 6H are schematic diagrams of interfaces for manually selecting a collaborative mode according to an embodiment of the present application;

[0055] 7 and 8 are schematic diagrams of interfaces for automatically switching collaborative modes according to embodiments of the present application;

[0056] 9A to 9D are schematic diagrams of interfaces for manually switching collaborative modes according to an embodiment of the present application;

[0057] FIG9E is a software implementation process provided in an embodiment of the present application;

[0058] 9F to 9I are schematic diagrams of interfaces for manually switching collaborative modes according to an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following is a clear and detailed description of the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0060] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more. The "upper", "lower", "left" and "right" involved in the embodiments of the present application are relative concepts and should not constitute a limitation on the embodiments of the present application.

[0061] The term "user interface (UI)" in the following embodiments of this application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that the user can recognize. The commonly used form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device. The user interface can also be referred to as an interface or page.

[0062] After a user's multiple terminal devices (such as mobile phones and laptops) establish a communication connection, they can collaboratively implement various collaborative modes, such as input device sharing, multi-screen collaboration, etc. At present, the application scenarios of various collaborative modes are relatively limited. For example, multi-screen collaboration (such as screen projection, application continuation) is usually applied to mobile phones and large-screen devices (such as personal computers (PCs), TVs), and input device sharing is usually applied to large-screen devices and large-screen devices (such as PCs and tablet computers (PADs)); due to limited communication resources (for example, input device sharing and screen projection both use WiFi P2P for communication), each terminal device usually only implements one of the collaborative modes of input device sharing and multi-screen collaboration at the same time. The user needs to manually control the terminal device to turn on the required collaborative mode, and the operation of switching collaborative modes is relatively cumbersome. For example, to switch the input device sharing between the PAD and the PC to screen projection, the user needs to open the input device sharing setting interface on the PC, and control the PC to turn off the input device sharing for the PAD through the setting interface; then, open the screen projection setting interface on the PAD, and control the PAD to project the screen to the PC through the setting interface. The multi-device collaboration method provided in the embodiment of the present application is not limited to the application scenarios of the collaborative mode (for example, mobile phones and large-screen devices can perform multi-screen collaboration and share input devices). After the two terminal devices establish a communication connection, the collaborative mode can be appropriately selected to match the user's real-time needs as much as possible.

[0063] The following introduces the communication system 10 involved in the multi-device collaboration method provided in an embodiment of the present application.

[0064] FIG1 exemplarily shows the system architecture of a communication system 10 provided in an embodiment of the present application. As shown in FIG1 , the communication system 10 includes at least two terminal devices, for example, a terminal device 100 and a terminal device 200 .

[0065] Among them, terminal device 100 and terminal device 200 have display screens. Terminal device 100 and terminal device 200 can be mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, and can also be cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices (smart bracelets), vehicle-mounted devices, smart home devices (smart TVs, smart screens, large-screen devices, etc.) and / or smart city devices. The device types of terminal device 100 and terminal device 200 can be the same or different; the embodiments of the present application do not impose any special restrictions on the specific types of terminal device 100 and terminal device 200. For example, the terminal device 100 is a mobile phone and the terminal device 200 is a laptop computer; or, the terminal device 100 is a tablet computer and the terminal device 200 is a mobile phone; or, the terminal device 100 is a tablet computer and the terminal device 200 is a laptop computer.

[0066] In an embodiment of the present application, the terminal device 100 and the terminal device 200 can establish a communication connection; based on the communication connection and multi-device collaboration technology, the terminal device 100 and the terminal device 200 can collaborate across systems and devices to achieve resource sharing and collaborative operation. The above-mentioned communication connection can also be referred to as a collaborative connection. With the continuous development of terminal technology, multi-device collaboration technology can support a variety of different collaboration modes, such as: input device sharing, multi-screen collaboration (such as screen projection, application continuation, etc.) and other collaboration modes. The above-mentioned collaboration mode can also be referred to as a collaborative function or other names, which are not specifically limited here.

[0067] In the embodiments of the present application, when terminal device 100 and at least one terminal device (e.g., terminal device 200) collaborate across systems and devices to achieve resource sharing and collaborative operations, the terminal device 100 and at least one terminal device (e.g., terminal device 200) can be considered a "hyperterminal," and a hyperterminal page can be provided on terminal device 100 and / or terminal device 200. Taking terminal device 200 as an example, the hyperterminal page on terminal device 200 is used to implement communication connections, task flow, collaborative modes, etc. between terminal device 200 and other terminal devices (e.g., terminal device 100). The hyperterminal page will be described in detail in subsequent embodiments and will not be repeated here.

[0068] Input device sharing: Referring to Figure 1, the terminal device 200 may have one or more input devices, such as a mouse and a keyboard; based on the above-mentioned communication connection, the terminal device 200 may share the above-mentioned one or more input devices with the terminal device 100, that is, the user may control the terminal device 100 through the above-mentioned one or more input devices of the terminal device 200. For example, by moving the mouse of the terminal device 200, the user may control the cursor corresponding to the mouse to cross the boundary of the screen of the device to move to the screen of the terminal device 100, and then may manipulate the display content of the terminal device 100 through the mouse; when the mouse selects the text input box displayed on the terminal device 100, the user may enter text in the text input box through the keyboard of the terminal device 200. The embodiment of the present application does not specifically limit the implementation principle of input device sharing. The embodiment of the present application does not specifically limit the display form of the cursor corresponding to the input device (such as a mouse or touchpad) on the screen, and the cursor may also be invisible.

[0069] In the embodiments of the present application, an input device is a device that inputs data and information into an electronic device. An input device is one of the main devices for exchanging information between a user and an electronic device, including but not limited to a mouse, keyboard, camera, tablet, stylus, remote control, touch panel, voice input device, etc. The input device can generate input data based on detected user operations and input it into the electronic device. The generated input data can indicate the above-mentioned user operations (such as click operations, movement operations), or can be information such as text, graphics, images, and sounds. The embodiments of the present application do not impose any restrictions on the type of input device and the input data generated by the input device. Referring to Figure 1, the input device can be an external device of the terminal device 200, connected to the terminal device 200 by wire or wirelessly. For example, a tablet computer can be wirelessly connected to a mouse; a laptop computer can be wired to a mouse. The input device can also be a built-in module of the terminal device 200, that is, a hardware module of the terminal device 200 itself. For example, the hardware modules of a laptop computer include a keyboard and a touch panel. The touchpad is used to control the cursor based on the input operations (such as movement and clicking) sensed by the user's fingers. The touchpad can be regarded as a substitute for a mouse. The embodiments of the present application do not specifically limit the type of input device. It should be noted that in some usage scenarios, the terminal device 100 does not have an input device; in some usage scenarios, the terminal device 100 may also have an input device, and the terminal device 100 may also share the input device with the terminal device 200.

[0070] Screen projection: Based on the above-mentioned communication connection, terminal device 100 sends the preset screen display content to terminal device 200 for display. Screen projection can include mirrored screen projection and non-mirrored screen projection. Mirrored screen projection refers to terminal device 100 sending the real-time displayed screen content to terminal device 200 for display; after projection, the display content of the projection window of terminal device 200 is consistent with the screen content displayed on terminal device 100. Non-mirrored screen projection refers to terminal device 100 sending the current screen display content or the display content of a specific application to terminal device 200; after projection, the display content of the projection window of terminal device 200 may be different from the screen content displayed on terminal device 100. Optionally, after mirrored screen projection and non-mirrored screen projection, the user can manipulate the display content from terminal device 100 in the projection window of terminal device 200, that is, can control terminal device 100 through the projection window. The embodiments of this application do not specifically limit the implementation principle of screen projection. It should be noted that in some usage scenarios, terminal device 200 can also project its screen to terminal device 100.

[0071] In some screen projection scenarios, the terminal device 100 projects the user interface of the currently displayed application 1 to the terminal device 200. The projection content only includes the user interface of the application, and the user can only control the application through the projection window displayed by the terminal device 100.

[0072] In some screen projection scenarios, the terminal device 100 can project the currently displayed system interface (such as the main interface) or the user interface of an application (such as the chat page of an instant messaging application) to the terminal device 200, and the terminal device 200 displays the projected content in the projection window; then, the user can open other user interfaces of the terminal device 100 through the projection window, such as other system interfaces (such as the negative one screen) and other application user interfaces (such as the user interface of text messages). It can be understood that the projected content in this scenario can include both system interfaces and application interfaces, and the user can freely control the terminal device 100 through the projection window.

[0073] Application Continuation: Based on the above-mentioned communication connection, terminal device 100 can transfer the currently running application 2 to terminal device 200 for continued display. Specifically, terminal device 100 can send the real-time usage data of application 2 to terminal device 200; based on the above-mentioned real-time usage data, terminal device 200 can locate the user's latest browsing location of application 2 on terminal device 100, so that terminal device 200 can continue to display application 2. Application 2 of terminal device 100 and terminal device 200 and application 2 can be different versions of application 2. For example, a mobile phone sends the real-time usage data of the mobile version of a video application to a laptop. The real-time usage data indicates the video page 1 that the user is playing in the video application and the current playback time of video 1 on video page 1, such as playback time 1 of video 1; the laptop runs the computer version of video application 1 and automatically starts playing video 1 on video page 1 from playback time 1 of video 1 based on the above-mentioned real-time usage data. The embodiment of the present application does not specifically limit the implementation principle of application continuity. It should be noted that in some usage scenarios, terminal device 100 can also perform application continuity for applications running on terminal device 200.

[0074] In an embodiment of the present application, different collaborative modes correspond to different start-up conditions, for example, input device sharing corresponds to start-up condition 1, and multi-screen collaboration corresponds to start-up condition 2. When the user initially turns on the collaborative mode, the terminal device 100 / terminal device 200 detects that the device status of the terminal device 100 currently satisfies the start-up condition, and then triggers the terminal device 200 and the terminal device 100 to start the collaborative mode corresponding to the start-up condition. In an embodiment of the present application, the user can trigger the start of any collaborative mode through the terminal device 100, or can trigger the start of any collaborative mode through the terminal device 200, and no specific limitation is made here. In this way, the user does not need to manually select the collaborative mode, and the collaborative mode can be adaptively selected according to the device status of the terminal device 100 to match the user's real-time needs.

[0075] In some embodiments, based on the above-mentioned communication connection, when the terminal device 100 and the terminal device 200 are in the first collaborative mode, if the terminal device 100 / terminal device 200 detects that the device status of the terminal device 100 meets the start-up conditions of the second collaborative mode, the first collaborative mode can be automatically switched to the second collaborative mode, that is, the first collaborative mode is stopped and the second collaborative mode is started. In some embodiments, based on the above-mentioned communication connection, when the terminal device 100 and the terminal device 200 are in the first collaborative mode, the user can also manually switch the first collaborative mode to the second collaborative mode. The embodiment of the present application does not specifically limit the operation of manual switching by the user. In the above embodiment, the first collaborative mode and the second collaborative mode cannot coexist.

[0076] In some embodiments, based on the above-mentioned communication connection, when the terminal device 100 and the terminal device 200 are in the first collaborative mode, if the terminal device 100 / terminal device 200 detects that the terminal device 100 meets the start-up conditions of the second collaborative mode, the second collaborative mode can be started, that is, the first collaborative mode and the second collaborative mode are performed at the same time. In some embodiments, based on the above-mentioned communication connection, when the terminal device 100 and the terminal device 200 are in the first collaborative mode, the user can also manually start the second collaborative mode. The embodiment of the present application does not specifically limit the operation manually started by the user. In the above embodiment, the first collaborative mode and the second collaborative mode can coexist.

[0077] For example, the first collaborative mode is input device sharing, and the second collaborative mode is multi-screen collaboration; or, the first collaborative mode is multi-screen collaboration, and the second collaborative mode is input device sharing.

[0078] It should be noted that the embodiment of the present application does not limit the type of the above-mentioned communication connection, and the terminal device 100 and the terminal device 200 can transmit data or instructions through the established communication connection.

[0079] In some embodiments, the terminal device 100 and the terminal device 200 can be directly connected via a short-range wireless communication connection or a local wired connection.

[0080] Exemplarily, the short-range communication modules involved in the above-mentioned short-range wireless communication connection may include a wireless local area network (WLAN) communication module, an ultra wide band (UWB) communication module, an infrared communication module, a Bluetooth communication module, a near field communication (NFC) communication module, a ZigBee communication module, etc. Among them, the Bluetooth module can provide solutions including one or more Bluetooth communications in classic Bluetooth (Bluetooth 2.1) or Bluetooth low energy (BLE), and the WLAN module can provide solutions including one or more WLAN communications in wireless fidelity peer-to-peer (Wi-Fi P2P), wireless fidelity local area networks (Wi-Fi LAN) or wireless fidelity software access point (Wi-Fi softAP). In some embodiments, Wi-Fi P2P refers to allowing devices in a wireless network to connect to each other in a point-to-point manner without going through a wireless router. The system is also called wireless fidelity direct (Wi-Fi direct). In one implementation, the terminal device 100 can transmit signals through a short-range communication module (such as a Bluetooth communication module) to detect and scan electronic devices near the terminal device 100, so that the terminal device 100 can discover nearby electronic devices (such as the terminal device 200) through a short-range wireless communication protocol, establish a short-range wireless communication connection (such as a Bluetooth connection) with the nearby electronic devices, and transmit data to the nearby electronic devices.

[0081] Exemplarily, the local wired connection may include a universal serial bus (USB) connection, a high definition multimedia interface (HDMI) connection, a display port (DP) connection, and the like.

[0082] In some embodiments, the terminal device 100 and the terminal device 200 can perform indirect communication over a long distance through at least one electronic device in a communication network, and the communication network includes a local area network (LAN) and / or a wide area network (WAN). In one implementation, the terminal device 100 and the terminal device 200 can be connected to the local area network (LAN) through at least one electronic device 300 based on a wired or wireless fidelity (WiFi) connection. For example, the electronic device 300 may include a third-party device such as a router, a gateway, or an intelligent device controller. In one implementation, the terminal device 100 and the terminal device 200 can also be indirectly connected through at least one electronic device 400 in a wide area network (such as the Internet). For example, the electronic device 400 may include one or more hardware servers, cloud servers embedded in a virtualized environment, and other devices. It can be understood that through the electronic device 300 and / or the electronic device 400, the terminal device 100 and the terminal device 200 indirectly perform wireless communication connection and data transmission.

[0083] In the embodiments of the present application, terminal device 100 and terminal device 200 can log in to the same account and then establish a communication connection via the Internet based on the same account. Alternatively, terminal device 100 and terminal device 200 can log in to different accounts but connect through binding. For example, a mobile phone and a laptop can log in to different accounts. The mobile phone can bind the laptop to itself in a device management application, and then control the communication connection through the device management application.

[0084] In some embodiments, when two terminal devices are mutually trusted (e.g., they have been paired or connected before), the two terminal devices can automatically establish a communication connection and then exchange data, eliminating the need for the user to manually reconnect or pair, saving time and effort. For example, electronic devices that have paired with Bluetooth, connected to a shared hotspot, and established a Wi-Fi P2P connection can be confirmed as trusted devices.

[0085] In some embodiments, input device sharing and multi-screen collaboration use the same communication resources (eg, WiFi P2P communication), and the terminal device 100 and the terminal device 200 can only implement one of input device sharing and multi-screen collaboration at the same time.

[0086] In some embodiments, input device sharing and multi-screen collaboration can use different communication resources, and terminal devices 100 and 200 can implement input device sharing and multi-screen collaboration simultaneously. For example, input device sharing uses WiFi P2P communication, and multi-screen collaboration uses Bluetooth communication; or, input device sharing and multi-screen collaboration use Bluetooth communication in a time-sharing manner; or, input device sharing and multi-screen collaboration use Bluetooth communication in a frequency-sharing manner.

[0087] Each terminal device in the communication system 10 can be equipped with system, system, system, HarmonyOS or other types of operating systems. The operating systems of each terminal device can be the same or different, and this application does not limit this.

[0088] It should be understood that FIG1 is merely a schematic diagram of the system structure of the communication system provided in the embodiment of the present application, and does not constitute a specific limitation on the communication system 10. In actual service scenarios, the communication system 10 may include more or fewer devices than shown in the figure.

[0089] The following describes the structure of a terminal device 100 provided in an embodiment of the present application. The structure of the terminal device 200 can refer to the relevant description of the terminal device 100 and will not be repeated hereafter.

[0090] 2A shows a schematic structural diagram of a terminal device 100. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0091] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0092] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0093] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0094] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0095] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0096] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C bus lines. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, and the like via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via the I2C interface, enabling communication between the processor 110 and the touch sensor 180K via the I2C bus interface, thereby implementing the touch function of the terminal device 100.

[0097] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of answering calls through a Bluetooth headset.

[0098] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0099] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the function of playing music through Bluetooth headphones.

[0100] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 100.

[0101] The GPIO interface can be configured via software. The GPIO interface can be configured as either a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0102] The USB interface 130 is an interface that complies with USB standards and specifications, and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the terminal device 100, or to transfer data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as AR devices.

[0103] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt a different interface connection method from the above embodiments, or a combination of multiple interface connection methods.

[0104] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the terminal device 100. While charging the battery 142, the charging management module 140 can also power the electronic device through the power management module 141.

[0105] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0106] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0107] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0108] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0109] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0110] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0111] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0112] The terminal device 100 implements display functions through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0113] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, terminal device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0114] The terminal device 100 can realize the shooting function through the ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0115] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise and brightness. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0116] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0117] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the terminal device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0118] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. This allows terminal device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0119] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in the terminal device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0120] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0121] Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation DDR SDRAM is generally referred to as DDR5 SDRAM), etc.; non-volatile memory may include disk storage devices and flash memory.

[0122] Flash memory can be divided into NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle; single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. according to the storage cell potential level; universal flash storage (UFS) and embedded multi media card (eMMC) can be divided into UFS and eMMC according to the storage specification.

[0123] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0124] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0125] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the terminal device 100. The external non-volatile memory communicates with the processor 110 via the external memory interface 120 to implement data storage. For example, files such as music and videos can be stored in the external non-volatile memory.

[0126] The terminal device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0127] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0128] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0129] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the user can hear the voice by placing the receiver 170B close to the ear.

[0130] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize sound signal collection, noise reduction, and can also identify the source of sound, realize directional recording function, etc.

[0131] The earphone jack 170D is used to connect a wired earphone and can be the USB interface 130 or a 3.5mm open mobile electronic device platform standard interface or a Cellular Telecommunications Industry Association standard interface.

[0132] The pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0133] The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., X, Y, and Z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for shooting anti-shake. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0134] It should be noted that the reference coordinate system of the gyroscope sensor is usually a geographic coordinate system. It is a geographic coordinate system, and the three-axis (i.e., X-axis, Y-axis, and Z-axis) coordinate system shown in FIG2B is a geographic coordinate system exemplified in an embodiment of the present application. As shown in FIG2B , the origin O of the geographic coordinate system can be located at the point where the carrier (i.e., the device containing the gyroscope sensor, such as the terminal device 100) is located, the X-axis points to the east (E) along the local latitude, the Y-axis points to the north (N) along the local meridian, and the Z-axis points upward along the local geographic vertical line, and forms a right-handed rectangular coordinate system with the X-axis and the Y-axis. Among them, the plane formed by the X-axis and the Y-axis is the local horizontal plane, and the plane formed by the Y-axis and the Z-axis is the local meridian plane.

[0135] For example, taking a smartphone as an example, the three-axis (i.e., Xs axis, Ys axis, and Zs axis) coordinate system shown in FIG2C is a device coordinate system of the terminal device 100 shown in an embodiment of the present application, wherein the origin of the device coordinate system can be a preset point (e.g., the center of mass) of the terminal device 100, the Xs axis points from the center of mass of the terminal device 100 to the right side of the terminal device, the Ys axis points from the center of mass of the terminal device 100 to the top of the terminal device 100, the Ys axis is perpendicular to the Xs axis, and the plane formed by the Xs axis and the Ys axis is parallel to the screen of the terminal device 100; and the Zs axis points from the center of mass of the terminal device to the screen of the terminal device, and is perpendicular to the plane formed by the Xs axis and the Ys axis.

[0136] The posture of the terminal device 100 can be determined by three posture angles: pitch, azimuth, and roll. In one implementation, the pitch angle can be the angle between the Ys axis of the device coordinate system and the local horizontal plane; the azimuth angle can be the angle between the projection of the Ys axis of the device coordinate system on the local horizontal plane and the Y axis of the ground coordinate system; the roll angle can be the angle between the Zs axis of the device coordinate system and the vertical plane of the Ys axis, and the vertical plane of the Ys axis can be understood as the plane formed by the Ys axis and the Z axis of the geographic coordinate system. In the embodiment of the present application, the terminal device 100 can determine the three posture angles of the terminal device 100 based on the data collected by the gyroscope, and then determine the current posture of the terminal device 100. In the embodiment of the present application, the method of determining the posture of the terminal device 100 is not specifically limited.

[0137] In an embodiment of the present application, the collaborative mode currently applicable to the terminal device 100 and the terminal device 200 can be determined based on the posture of the terminal device 100. Taking a mobile phone and a laptop as an example, when the screen of the mobile phone is not tilted upright or facing the user, it is not convenient for the user to view the screens of the mobile phone and the laptop at the same time, and it is not convenient to control the screen of the mobile phone through the mouse of the laptop, that is, it is not convenient to share input devices. In this case, multi-screen collaboration can be adopted. Correspondingly, when the mobile phone is tilted upright and the screen of the mobile phone faces the user, it is convenient for the user to view the screens of the mobile phone and the laptop at the same time, and it is convenient to control the screen of the mobile phone through the mouse of the laptop, that is, it is convenient to share input devices.

[0138] In an embodiment of the present application, whether the posture of the terminal device 100 is inclined toward upright can be determined based on data collected by the gyroscope sensor 180B. In one implementation, the terminal device 100 being inclined toward upright means: the pitch angle of the terminal device 100 is greater than angle threshold 1 (e.g., 60°) and less than or equal to angle threshold 2 (e.g., 100°), and the absolute value of the angle between the Xs axis of the terminal device 100 and the horizontal plane is less than or equal to angle threshold 3 (e.g., 15°); or the roll angle of the terminal device 100 is greater than angle threshold 1 (e.g., 60°) and less than or equal to angle threshold 2 (e.g., 100°), and the absolute value of the angle between the Ys axis of the terminal device 100 and the horizontal plane is less than or equal to angle threshold 3 (e.g., 15°). In another implementation, the terminal device 100 leaning toward upright means that: the angle between the screen of the terminal device 100 (i.e., the plane formed by the Xs axis and Ys axis of the terminal device 100) and the horizontal plane is greater than an angle threshold 4 (e.g., 60°) and less than or equal to an angle threshold 5 (e.g., 100°), and the absolute value of the angle between the Xs axis or the Ys axis of the terminal device 100 and the horizontal plane is less than or equal to an angle threshold 6 (e.g., 15°). This embodiment of the present application does not specifically limit the implementation method for detecting whether the posture of the mobile phone leans toward upright.

[0139] For ease of description, this application refers to the screen of the terminal device 100 being in a posture that is convenient for the user to manipulate through the input device as a convenient posture. For example, the convenient posture means that the terminal device 100 is inclined to be upright, and the screen of the terminal device 100 faces the user.

[0140] In some embodiments, when the terminal device 200 shares an input device with the terminal device 100, if the user changes the posture of the terminal device 100 so that the screen of the terminal device 100 is out of the convenient posture, the terminal device 100 and the terminal device 200 can automatically switch the input device sharing to multi-screen collaboration (such as screen projection, application continuation) without user operation. In this way, when it is inconvenient for the user to control the terminal device 100 through the input device of the terminal device 200, the user can continue to view and control the display content from the terminal device 100 through the screen projection window or application continuation window in a timely manner.

[0141] Conversely, in some embodiments, when terminal device 100 and terminal device 200 are performing multi-screen collaboration, if the user changes the posture of terminal device 100 so that the screen of terminal device 100 switches to a convenient posture, without user operation, terminal device 100 and terminal device 200 can automatically switch the multi-screen collaboration to input device sharing. In this way, when the screen of terminal device 100 switches to a posture that is convenient for control through input devices, the user can also view and control terminal device 100 through the input devices of terminal device 200 in a timely manner.

[0142] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

[0143] The magnetic sensor 180D includes a Hall sensor, and the terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case.

[0144] Accelerometer 180E can detect the magnitude of acceleration of the terminal device 100 in all directions (generally three axes). When the terminal device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0145] The distance sensor 180F is used to measure distance. The terminal device 100 can measure distance using infrared or laser. In some embodiments, when shooting a scene, the terminal device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0146] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light outward through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can use the proximity light sensor 180G to detect when the user holds the terminal device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0147] In an embodiment of the present application, when an anti-mistouch scenario is detected (for example, a scenario where the terminal device 100 is placed in a pocket or bag), the terminal device 100 can turn on the anti-mistouch function to prevent the user from accidentally touching the screen of the terminal device 100. If the user accidentally touches the screen, the device may be mistakenly locked or the application may be accidentally clicked, thereby causing erroneous operations, such as making a wrong call, deleting a text message, or logging into the network and using traffic. In severe cases, it may cause the loss of important information. Turning on the anti-mistouch function in the anti-mistouch scenario can provide the user with a friendly operating environment and reduce the power consumption of the device. In the embodiment of the present application, the above-mentioned anti-mistouch scenario and the method of detecting the anti-mistouch scenario are not specifically limited; in addition, the user can also manually turn on the anti-mistouch function of the terminal device 100.

[0148] In some embodiments, the proximity light sensor 180G can be used to determine whether the anti-accidental touch function can be enabled. For example, when the proximity light sensor 180G detects that the screen of the terminal device 100 is obstructed within a preset distance, the terminal device 100 enables the anti-accidental touch function. The embodiments of the present application can also be combined with other sensors (such as ultrasonic sensors) to detect whether the screen of the terminal device 100 is obstructed within a preset distance, which is not specifically limited here.

[0149] The ambient light sensor 180L is used to sense the brightness of the ambient light. The terminal device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in a pocket or bag to prevent accidental touch. In one embodiment, the ambient light sensor 180L and the proximity light sensor 180G can be coupled to determine whether the terminal device 100 has turned on the anti-false touch function. For example, when the ambient light sensor 180L detects that the ambient light brightness is lower than the first brightness value, such as 10 lux (lx), and the proximity light sensor 180G detects that the screen of the current terminal device 100 is in an obscured state, the terminal device 100 will turn on the anti-false touch function.

[0150] In some embodiments, when the terminal device 200 shares the input device with the terminal device 100, if it is detected that the terminal device 100 is in an anti-mistouch scenario or the anti-mistouch function is turned on, no user operation is required, and the terminal device 100 and the terminal device 200 can automatically switch the input device sharing to multi-screen collaboration.

[0151] In some embodiments, when the terminal device 100 is projecting its screen to the terminal device 200, if it is detected that the terminal device 100 has left the anti-mistouch scenario or has turned off the anti-mistouch function, the terminal device 100 and the terminal device 200 can automatically switch the multi-screen collaboration to input device sharing without user operation.

[0152] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0153] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy.

[0154] The touch sensor 180K is also called a "touch control device." The touch sensor 180K can be provided on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be provided on the surface of the terminal device 100, at a location different from that of the display screen 194.

[0155] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone mass in a human vocal part.

[0156] The buttons 190 include a power button, a volume button, etc. The buttons 190 may be mechanical buttons or touch buttons. The terminal device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the terminal device 100.

[0157] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts and can also be used for touch vibration feedback.

[0158] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0159] The SIM card interface 195 is used to connect a SIM card.

[0160] For example, taking the terminal device 100 equipped with HarmonyOS as an example, FIG3 shows a software system architecture of the terminal device 100 .

[0161] As shown in Figure 3, HarmonyOS 200 adopts a layered architecture as an example. The layered architecture divides HarmonyOS into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, HarmonyOS is divided into four layers, from top to bottom, namely the application layer, the application framework layer, the system service layer, and the kernel layer. In other embodiments, HarmonyOS 200 can also be divided into other number of hierarchical structures, which is not limited here.

[0162] Among them, the application layer can include a series of applications (Application, APP) such as system applications and third-party applications (or extended applications). System applications may include applications installed by default on terminal devices such as the desktop, control bar, settings, and phone. Extended applications can be non-essential applications developed and designed by the manufacturer of the terminal device, such as device managers, music, notes, instant messaging applications, and other applications. Third-party non-system applications can be developed by other manufacturers, but can run applications in the Harmony system, such as games, navigation, social or shopping applications.

[0163] In some embodiments, the applications in the application layer of the terminal device 100 may include two categories of applications. The first category is applications that can be connected to the terminal device 200. The applications are applications that can be installed by both the terminal device 100 and the terminal device 200. They can be system applications or third-party applications, such as video, navigation, and instant messaging. The second category is applications that cannot be connected to the terminal device 200, such as text messages, phone calls, desktops, etc. in system applications, or third-party applications that cannot be installed by the terminal device 200.

[0164] In some embodiments of the present application, applications can be atomized. Specifically, applications can be composed of one or more capabilities (also known as atomic capabilities, etc.). Capabilities include but are not limited to feature capabilities (FA, also known as meta-programs) and meta-capabilities (AA, also known as meta-services). FA has a user interface (UI) and provides the ability to interact with users; AA has no UI interface and provides the ability to run tasks in the background and a unified data access abstraction. FA and AA can also have other names, which are not limited in the embodiments of the present application.

[0165] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. For example, it provides a user interface (UI) framework, a capability framework, and a user program framework.

[0166] The system service layer is the core capability set of the system, providing services to the applications in the application layer through the framework layer. This layer includes the following parts:

[0167] Basic capability subsystem set: provides basic capabilities for the operation, scheduling, migration, and other operations of distributed applications on multiple devices. It can include distributed soft bus (or, it can also be described as soft bus subsystem), distributed data management module, distributed task scheduling module, public basic library (not shown in the figure), multi-mode input, graphics, security, artificial intelligence (AI) and other subsystems.

[0168] Among them, the distributed soft bus is a logical channel that can be used for communication between devices. In the embodiment of the present application, multiple electronic devices (such as terminal device 100 and terminal device 200) can achieve communication connection, data transmission, hardware mutual assistance, and resource sharing through the distributed soft bus, distributed device virtualization, distributed data management, and distributed task scheduling. For example, collaboratively realize input device sharing, screen projection, and application continuation.

[0169] The multi-mode input subsystem provides input services for the terminal device 100, monitors the user's input operations on the terminal device 100, such as receiving input events through input devices such as a touch screen, mouse, and keyboard, and processes these raw input events before dispatching them. In some embodiments, when the input device of the terminal device 200 is shared with the terminal device 100, the multi-mode input subsystem of the terminal device 200 can be used to monitor the input events (such as input event 1) of the input device of the terminal device 200 (such as a mouse) to the terminal device 100, and can share input event 1 with the terminal device 100; the multi-mode input subsystem of the terminal device 100 can process and distribute the monitored input event 1; in this way, the user can control the terminal device 100 through the input device of the terminal device 200.

[0170] In some embodiments, after the input device of the terminal device 200 is shared with the terminal device 100, when the multi-mode input subsystem of the terminal device 200 monitors the input event 1 of the terminal device 100 by the input device of the terminal device 200 (such as a mouse), it is shared with the multi-mode input subsystem of the terminal device 100 through the distributed soft bus, and the multi-mode input subsystem of the terminal device 100 processes and distributes the input event 1.

[0171] In some embodiments, the distributed data management module can be used to receive and record application status data, and synchronize the application status data to other terminal devices in the distributed system (such as the communication system shown in Figure 1) after the application status changes.

[0172] In some embodiments, after the input device of the terminal device 200 is shared with the terminal device 100, when the multi-mode input subsystem of the terminal device 200 monitors the input event 1 of the input device of the terminal device 200 (such as a mouse) for the terminal device 100, the input event 1 can be uploaded to the distributed data management module, and then the input event 1 can be shared with the distributed data management module of the terminal device 100; the multi-mode input subsystem of the terminal device 100 monitors the input event 1 from the distributed data management module, and processes and distributes the input event 1.

[0173] The drag framework can be used to enable, in input device sharing mode, the dragging of images, files, user interfaces, and other content from the screen of the terminal device 200 to the screen of the present device via an input device such as a mouse, or vice versa. For example, when the user interface of the present device is dragged to the screen of the terminal device 200, the drag framework of the present device can obtain an interface snapshot of the interface and send it directly to the drag framework of the terminal device 200 via the distributed soft bus, or share it with the drag framework of the terminal device 200 via the distributed data management module; the drag framework of the terminal device 200 can trigger the display of the mouse dragging the interface snapshot to move on the screen of the terminal device 200.

[0174] The crossing service is used to sense mouse events such as pressing the mouse, moving the mouse, and releasing the mouse, and to process the above mouse events to enable the cursor corresponding to the mouse to cross between the screen of the terminal device 200 and this device.

[0175] Basic software service subsystem set: provides public and general software services for the system, consisting of subsystems such as event notification, telephone, multimedia, DFX, MSDP&DV, etc.

[0176] Enhanced software service subsystem set: Provides the system with differentiated capability-enhanced software services for different devices, consisting of subsystems such as smart screen proprietary services, wearable proprietary services, and IoT proprietary services.

[0177] Hardware service subsystem set: provides hardware services and consists of subsystems such as location services, biometric recognition, wearable proprietary hardware services, and IoT proprietary hardware services.

[0178] According to the deployment environment of different device forms, the basic software service subsystem set, enhanced software service subsystem set, and hardware service subsystem set can be tailored according to the subsystem granularity, and each subsystem can be tailored according to the functional granularity.

[0179] The kernel layer is the layer between hardware and software. The kernel layer is divided into the kernel subsystem and the driver subsystem.

[0180] Kernel subsystem: Adopts a multi-core design. The kernel abstraction layer (KAL) on the kernel subsystem shields the differences between multiple cores and provides basic kernel capabilities to the upper layer, including process / thread management, memory management, file system, network management, and peripheral management.

[0181] The driver subsystem is the system's driver framework (HDF) and the foundation for an open distributed system hardware ecosystem. It provides unified peripheral access capabilities and a driver development and management framework. The kernel layer includes at least display drivers, camera drivers, audio drivers, and sensor drivers.

[0182] The system architecture shown in FIG3 does not constitute a limitation on the system architecture of the terminal device. The system architecture of the terminal device 100 may have more or fewer system architecture structures than the structure shown in FIG3 , or may be other forms of system architecture, which is not specifically limited here.

[0183] In combination with the aforementioned communication system, hardware structure, and software system architecture, embodiments of the present application provide a multi-device collaboration method in which terminal device 100 and terminal device 200 can select and implement an appropriate collaboration mode from a variety of collaboration modes based on the real-time status of terminal device 100 and / or user operations, thereby minimizing the user's real-time needs. These multiple collaboration modes may include: input device sharing, screen projection, and application continuation.

[0184] For ease of understanding, the following describes the multi-device collaboration method provided by the embodiments of the present application in detail in conjunction with interface embodiments 1 to 4. For ease of description, the subsequent embodiments are described using the example of a mobile phone as the terminal device 100 and a computer as the terminal device 200 as the terminal device 200.

[0185] Interface Example 1

[0186] In the first embodiment of the interface, different device states of the mobile phone 100 may correspond to different collaborative modes. When the collaborative mode is initially started, under different device states of the mobile phone 100, the user can trigger the mobile phone 100 and the computer 200 to enter different collaborative modes through a preset operation 1.

[0187] In some embodiments, after detecting a preset operation 1 acting on the mobile phone 100 / computer 200, the mobile phone 100 / computer 200 detects the device status of the mobile phone 100; when the device status of the mobile phone 100 meets the start condition 1, the mobile phone 100 / computer 200 can trigger the start of input device sharing; when the device status of the mobile phone 100 meets the start condition 2, the mobile phone 100 / computer 200 can trigger the start of multi-screen collaboration (such as screen casting, application continuation). It can be understood that in the embodiments of the present application, the initial startup of the collaborative mode can be triggered by the mobile phone 100, and the initial startup of the collaborative mode can also be triggered by the computer 200, and no specific limitation is made here.

[0188] Exemplarily, FIG. 4A to FIG. 4I show that a user triggers two terminal devices to start the first collaborative mode through the computer 200 .

[0189] 4A and 4B show the user interface of the computer 200 starting the hyperterminal page.

[0190] For example, FIG4A shows a main interface 11 of a computer 200, wherein the status bar of the main interface 11 includes a HyperTerminal icon 101. As shown in FIG4A and FIG4B , when the computer 200 detects an input operation on the HyperTerminal icon 101 (e.g., a click operation performed by the mouse when the mouse cursor moves to the HyperTerminal icon 101), the computer 200 displays a HyperTerminal page 12.

[0191] In the embodiment of the present application, the user may also open the HyperTerminal page 12 through other methods. For example, the user may open the HyperTerminal page 12 through the control center, notification bar, desktop card, or an application (such as a smart home application or settings application) of the computer 200. This embodiment does not limit the method of opening the HyperTerminal page 12. The HyperTerminal page 12 shown in Figure 4B is an example provided in the embodiment of the present application. The HyperTerminal page 12 may include more or fewer interface elements, which are not specifically limited here.

[0192] The hyperterminal page 12 may include a hyperterminal icon 102, a more icon 103, a zoom out icon 104, an enlargement icon 105, and a close icon 106. The more icon 103 is used to trigger more functions of the hyperterminal, the zoom out icon 104 is used to minimize the hyperterminal page 12, the enlargement icon 105 is used to maximize the hyperterminal page 12, and the close icon 106 is used to close the hyperterminal page 12.

[0193] The hyperterminal page 12 further includes an icon 107 of the computer 200 and an icon of at least one candidate device detected by the computer 200 for collaborative connection, such as an icon 108 of the mobile phone 100 .

[0194] For example, the icon 107 of the computer 200 is located in the preset area 1 (e.g., the circular area shown in FIG4B ) of the hyperterminal page 12, and the icon of the at least one candidate device can be located in the preset area 2 (e.g., the annular area shown in FIG4B ) surrounding the icon 107 of the computer 200. This embodiment of the application does not specifically limit the shape and position of the preset area 1 and the preset area 2. This embodiment of the application also does not specifically limit the distribution method of the at least one candidate device in the preset area 2.

[0195] In some embodiments, the distance between the icon 108 of the mobile phone 100 and the icon 107 of the computer 200 is determined based on the distance between the mobile phone 100 and the computer 200; the orientation of the icon 108 of the mobile phone 100 relative to the icon 107 of the computer 200 is determined based on the orientation of the mobile phone 100 relative to the computer 200. For example, when the mobile phone 100 is located on the left side of the computer 200, the icon 108 of the mobile phone 100 is also located on the left side of the icon 107 of the computer 200. In the embodiments of the present application, the method for measuring the relative distance and relative orientation between the computer 200 and the mobile phone 100 is not specifically limited.

[0196] In some embodiments, upon receiving and responding to a user's preset operation 1 on the icon 108 of the mobile phone 100 (e.g., dragging the icon 108 close to the icon 107 of the computer 200), the computer 200 requests a collaborative connection with the mobile phone 100, and detects the device status of the mobile phone 100 after the collaborative connection; if it is detected that the device status of the mobile phone 100 meets the startup condition, the computer 200 automatically instructs the mobile phone 100 to perform the collaborative mode corresponding to the startup condition.

[0197] In some embodiments, the hyperterminal page 12 may further include a prompt message 109, which is used to prompt that a collaborative connection can be established by dragging the device icon to the local icon. In some embodiments, referring to Figures 4B and 4C, after detecting that the user has selected the icon 108 of the mobile phone 100 (for example, by long pressing the left mouse button to select the icon 108), dragging the icon 108 close to the icon 107 of the computer 200, and stopping selecting the icon 108 (for example, by stopping long pressing the left mouse button to stop selecting the icon 108), the computer 200 adsorbs the icon 108 of the mobile phone 100 on the edge of the icon 107 of the computer 200 and establishes a collaborative connection with the mobile phone 100; after successfully establishing the collaborative connection, a prompt message may be displayed to remind the user that the mobile phone 100 and the computer 200 have established a collaborative connection. In one implementation, when the computer 200 detects that the distance between the icon 108 of the mobile phone 100 and the icon 107 of the computer 200 is less than or equal to distance 1, the computer 200 and the mobile phone 100 establish a collaborative connection.

[0198] In some embodiments, after computer 200 and mobile phone 100 establish a collaborative connection based on preset operation 1, computer 200 may send request message 1 to mobile phone 100. Request message 1 is used to instruct mobile phone 100 to detect whether the device status of mobile phone 100 meets the activation conditions corresponding to each collaborative mode. Taking the case of satisfying activation condition 1 as an example, when mobile phone 100 detects that the device status of mobile phone 100 meets activation condition 1 based on relevant parameters of the device status, it sends response message 1 to computer 200. Response message 1 indicates activation condition 1 or the collaborative mode corresponding to activation condition 1. Based on response message 1, computer 200 determines to instruct mobile phone 100 to initiate the collaborative mode corresponding to activation condition 1.

[0199] In some embodiments, based on preset operation 1, after the computer 200 and the mobile phone 100 establish a collaborative connection, the computer 200 can send a request message 2 to the mobile phone 100, and the request message 2 is used to obtain relevant parameters of the device status of the mobile phone 100; after the computer 200 receives the above-mentioned relevant parameters fed back by the mobile phone 100, it detects whether the device status of the mobile phone 100 meets the startup conditions corresponding to each collaborative mode based on the above-mentioned relevant parameters; when it is determined that the device status of the mobile phone 100 meets the startup conditions, the computer 200 determines to instruct the mobile phone 100 to perform the collaborative mode corresponding to the startup conditions.

[0200] In some embodiments, based on the preset operation 1, after the computer 200 and the mobile phone 100 establish a collaborative connection, the computer 200 can send a request message 3 to the mobile phone 100, and the request message 3 is used to instruct the mobile phone 100 to perform multi-device collaboration; when the mobile phone 100 detects that the device status of the mobile phone 100 meets the startup condition according to the request message 3, the mobile phone 100 determines to instruct the computer 200 to perform the collaborative mode corresponding to the startup condition.

[0201] In the embodiment of the present application, the developer can define the starting conditions of each collaborative mode according to the actual application scenario (such as the device type of the device performing multi-device collaboration), which is not specifically limited here.

[0202] In some embodiments, the computer 200 receives and responds to the preset operation 1. If it detects that the device status of the mobile phone 100 meets the start condition 1, the computer 200 shares the input device (such as a mouse) with the mobile phone 100, and the user can control the mobile phone 100 through the input device of the computer 200.

[0203] For example, as shown in FIG4C , if the device status of mobile phone 100 satisfies startup condition 1 and computer 200 shares its mouse with mobile phone 100, if the cursor follows the mouse and moves to the edge of the computer 200 screen and then continues to move outside the screen, the cursor can cross from the edge of the computer 200 screen to the edge of the mobile phone 100 screen and continue to move on the screen of mobile phone 100. At this time, the cursor is not displayed on the computer 200 screen, but is displayed on the mobile phone 100 screen. When the cursor moves to the screen of mobile phone 100, the user can control the mobile phone 100 to perform various operations using the mouse. As shown in FIG4C and FIG4D , the main interface 13 of mobile phone 100 includes an instant messaging application icon 111. After the cursor moves to icon 111, the mobile phone 100 receives and responds to the user's double-clicking of icon 111 with the mouse. The instant messaging application corresponding to icon 111 is run in the foreground of mobile phone 100, and the homepage 14 of the instant messaging application is displayed. Similarly, after the cursor crosses the screen of the mobile phone 100, if the cursor follows the mouse to the screen boundary of the mobile phone 100 and then continues to move toward the outside of the screen, the cursor can cross from the screen boundary of the mobile phone 100 back to the screen of the computer 200.

[0204] In some embodiments, following the movement of the mouse, the cursor can pass through boundary 1 of the screen of the computer 200 and pass through boundary 2 of the screen of the mobile phone 100. Boundary 1 and boundary 2 can be any one of the left boundary, right boundary, upper boundary, and lower boundary of the screen. In one implementation, "boundary 1 and boundary 2" are associated with the "relative position of the mobile phone 100 and the computer 200." For example, as shown in FIG4C , the mobile phone 100 is located on the left side of the computer 200. The user moves the mouse to the left, causing the cursor to pass through the left boundary of the screen of the computer 200 and pass through the right boundary of the screen of the mobile phone 100. The subsequent embodiments are illustrative in nature using the left boundary of the screen of the computer 200 and the right boundary of the screen of the mobile phone 100 as examples.

[0205] In some embodiments, the above-mentioned starting condition 1 includes one or more of the following: "not in an anti-mistouch scenario (for example, placed in a bag or pocket), the anti-mistouch function is not turned on, the screen is facing the user (for example, facing the desktop, facing the sky, facing away from the user), the posture tends to be upright, and the distance between the mobile phone 100 and the computer 200 is less than distance 2".

[0206] In some embodiments, the above-mentioned starting condition 1 includes: the posture of the mobile phone 100 tends to be upright, and the distance between the mobile phone 100 and the computer 200 is less than distance 2.

[0207] In one implementation, the mobile phone 100 stores a preset user's face; the front-facing camera of the mobile phone 100 captures an image 1. If the image 1 detects that the preset user's face is included, the screen of the mobile phone 100 is determined to be facing the user. In one implementation, the mobile phone 100 stores a preset user's face; the front-facing camera of the mobile phone 100 captures an image 1. If the image 1 detects that the preset user's face is included, and the angle between the face plane and the screen is determined to be less than an angle threshold 7 (e.g., 55°) based on the facial features of the user, the screen of the mobile phone 100 is determined to be facing the user. The face plane can be determined based on the facial features of the user.

[0208] It should be noted that the embodiments of the present application do not specifically limit the distance measurement method between the two devices, such as ultrasonic ranging algorithm, time-of-flight ranging algorithm, GPS positioning method, etc. The implementation method of detecting whether the posture of the mobile phone is inclined to be upright can refer to the above embodiments and will not be repeated here.

[0209] In some embodiments, the computer 200 shares the input device with the mobile phone 100, including: referring to FIG4C , the computer 200 sends a message 1 to the mobile phone 100, which indicates that the input device is shared with the mobile phone 100, the kernel layer of the computer 200 creates a virtual display screen corresponding to the mobile phone 100, and the mouse cursor of the computer 200 can be moved from the physical display screen to the above-mentioned virtual display screen; the multi-mode input subsystem of the mobile phone 100 adds the input device information (such as the device ID) corresponding to each input device shared by the computer 200 based on the message 1, and the multi-mode input subsystem can monitor the input events of the input device shared by the computer 200 according to the above-mentioned input device information. When the cursor moves to the virtual display screen corresponding to the mobile phone 100, the multi-mode input subsystem of the computer 200 obtains the input event of the shared input device (such as a mouse) and converts the above input event into the input event of the mobile phone 100 (such as a mouse movement event, a mouse click event) according to the event mapping table, and transmits it to the mobile phone 100; the multi-mode input subsystem of the mobile phone 100 can monitor the above input events transmitted by the computer 200, and process and distribute the above input events, and then the mobile phone 100 can respond to the above input events, such as displaying the cursor, moving the cursor, etc. In one implementation, the resolution of the above virtual display screen and the physical display screen of the mobile phone 100 are the same, and each pixel corresponds one to one, so the coordinates of the cursor on the virtual display screen are the same as the coordinates of the cursor on the mobile phone 100, and the coordinates of the cursor on the virtual display screen can be sent directly to the mobile phone 100 through the input event without the need for complex coordinate conversion. The embodiment of the present application does not specifically limit the implementation of input device sharing between the computer 200 and the mobile phone 100.

[0210] In some embodiments, the computer 200 receives and responds to a preset operation 1. If it is detected that the device status of the mobile phone 100 meets the start condition 2, the computer 200 instructs the mobile phone 100 to project the preset screen display content to the computer 200.

[0211] For example, as shown in Figure 4E , when the device status of mobile phone 100 satisfies start-up condition 2, mobile phone 100 uses screen projection technology to send the current main interface 13 to computer 200, which then displays the main interface 13 from mobile phone 100 in screen projection window 15. Optionally, computer 200 can manipulate the displayed content within the screen projection window. As shown in Figures 4E and 4F , the main interface 13 displayed in screen projection window 15 includes an instant messaging application icon 111. After the cursor moves to icon 111, the user double-clicks icon 111 with the mouse, and screen projection window 15 displays the homepage 14 of the instant messaging application corresponding to icon 111.

[0212] In some embodiments, the above-mentioned start condition 2 includes: the mobile phone 100 does not meet the start condition 1. In one implementation, the start condition 2 includes at least one of the following: the mobile phone 100 is in a false touch prevention scenario, the false touch prevention function is turned on, the screen is not facing the user and is not tilted upright, and the distance between the mobile phone 100 and the computer 200 is greater than or equal to distance 2.

[0213] In some embodiments, the above-mentioned start condition 2 includes: the mobile phone 100 does not meet the start condition 1, and the page running in the foreground of the terminal device 100 belongs to the application 1 that cannot be applied to the computer 200. The application 1 can be a system application (such as the desktop, the negative one screen, the control bar, etc.) or a third-party application. The mobile phone 100 has version 1 of application 1 installed (i.e., the mobile phone version of application 1); the computer 200 cannot apply the application 1 to continue, which means: there is no version 2 of application 1 suitable for the device type of the computer 200 (i.e., the computer version of application 1); or, there is a version 2 of application 1 suitable for the computer 200, but the computer 200 does not have this version of application 1 installed. It can be understood that version 1 suitable for mobile phones and version 2 suitable for computers are different versions; if the aforementioned terminal device 100 and terminal device 200 are of the same device type (such as a mobile phone), the versions of application 1 applicable to the two devices can be the same.

[0214] In some embodiments, the computer 200 instructs the mobile phone 100 to project the preset screen display content to the computer 200, including: the computer 200 sends a message to the mobile phone 100, the message being used to instruct the mobile phone 100 to project the screen to the computer 200; the mobile phone 100 creates a virtual display screen for projection, runs the user interface to be projected to the computer 200 (such as the main interface 13) on the virtual display screen, and sends image data of the display content of the virtual display screen to the computer 200; the computer 200 synchronously displays the display content of the virtual display screen in the projection window based on the above image data. When the multi-mode input subsystem of the computer 200 detects an input event in the projection window, the input event is converted into an input event of the virtual display screen of the mobile phone 100 and sent to the mobile phone 100; the multi-mode input subsystem of the mobile phone 100 can monitor the input event transmitted by the computer 200, process and distribute the input event, and then the mobile phone 100 updates the display content of the virtual display screen according to the input event and sends the updated display content to the computer 200; the computer 200 will display the updated display content in the projection window. The embodiments of the present application do not specifically limit the implementation method of screen projection.

[0215] In some embodiments, the computer 200 receives and responds to the preset operation 1. If it is detected that the device status of the mobile phone 100 meets the start condition 2, the computer 200 performs an application connection on the application running in the foreground of the mobile phone 100.

[0216] For example, as shown in FIG4G , the device status of mobile phone 100 satisfies startup condition 3. Mobile phone 100 runs chat page 16 of contact "Henry" in the instant messaging application in the foreground. Mobile phone 100 instructs computer 200 to connect to the instant messaging application and sends real-time usage data of the instant messaging application to computer 200. The real-time usage data may indicate the page identifier of chat page 16 and the user's browsing position within chat page 16. Based on the real-time usage data, computer 200 runs the instant messaging application, displays the homepage 18 of the instant messaging application in area 1 of application connection window 17, and displays chat page 19 of contact "Henry" in area 2 based on the browsing position. Computer 200 can control the displayed content within the application connection window. For example, as shown in Figures 4G and 4H, the home page 18 includes an option 112 for "Group Chat 1"; after the cursor moves to option 112, it receives and responds to an operation on option 112 (such as a left-click operation performed by the user with a mouse), and the computer 200 displays the chat page 20 of the contact's "Group Chat 1" in area 2 of the application connection window 17.

[0217] In some embodiments, the above-mentioned start condition 2 includes: the mobile phone 100 does not meet the start condition 1.

[0218] In some embodiments, the above-mentioned start condition 2 includes: the mobile phone 100 does not meet the start condition 1, and the page running in the foreground of the terminal device 100 belongs to application 2 that the computer 200 can apply to continue.

[0219] In some embodiments, the mobile phone 100 has version 1 of application 2 installed (i.e., mobile phone version of application 2); the computer 200 can apply the subsequent application 2, which means: there is a version 2 of application 2 suitable for the computer 200, and the computer 200 has this version of application 2 installed.

[0220] In some embodiments, referring to FIG4I , if it is detected that there is a version 2 of application 2 suitable for computer 200, but computer 200 does not have that version of application 2 installed, computer 200 may display a prompt message 113, which prompts whether to install version 2 of application 2. It may also display an installation control 114 and a cancel control 115 for version 2 of application 2. The installation control 114 is used to trigger the installation of version 2 of application 2; the cancel control 115 is used to cancel the installation of version 2 of application 2. As shown in FIG4I , if computer 200 detects that the user has successfully installed application 2 through the installation control 114, computer 200 will continue the application of application 2 running in the foreground of mobile phone 100; if computer 200 detects an operation on the cancel control 115 (e.g., a left-click operation performed by a mouse), computer 200 will instruct mobile phone 100 to cast its screen to computer 200.

[0221] In some embodiments, the computer 200 receives and responds to the preset operation 1. If it detects that the device status of the mobile phone 100 meets the starting condition 2, the computer 200 instructs the mobile phone 100 to perform multi-screen collaboration. If the page running in the foreground of the mobile phone 100 is an application that cannot be applied to the computer 200, the multi-screen collaboration refers to the screen projection of the mobile phone 100 to the computer 200; otherwise, the multi-screen collaboration refers to the computer 200 performing application succes on the application running in the foreground of the mobile phone 100.

[0222] For example, referring to Figures 5A and 5B , the mobile phone 100 may also be provided with a hyperterminal page. For example, Figure 5A illustrates a control center interface 21 of the mobile phone 100 provided in an embodiment of the present application. The control center interface 21 includes a hyperterminal card 116 . Upon receiving and responding to an operation on the card 116 , the mobile phone 100 may display a hyperterminal page 22 . The hyperterminal page 22 includes an icon 117 for the mobile phone 100 and an icon 118 for at least one terminal device that can establish a collaborative connection with the mobile phone 100, such as a computer 200 . For the specific implementation of the hyperterminal page 22 , please refer to the relevant description of the hyperterminal page 12 .

[0223] In one implementation, the mobile phone 100 receives and responds to a preset operation 1 for the icon 118 of the mobile phone 100 in the hyperterminal page 12 (for example, the user drags the icon 118 of the computer 200 close to the icon 117 of the mobile phone 100), and the mobile phone 100 requests to establish a collaborative connection with the computer 200 and detects the device status of the mobile phone 100; when it is detected that the device status of the mobile phone 100 meets the startup condition, the mobile phone 100 automatically instructs the computer 200 to enter the collaborative mode corresponding to the startup condition.

[0224] In some embodiments, the mobile phone 100 receives and responds to a preset operation 1. If the mobile phone 100 detects that the device status of the mobile phone 100 meets the start condition 1, it sends a message to the computer 200, which is used to instruct the computer 200 to share the input device with the mobile phone 100; if the mobile phone 100 detects that the device status of the mobile phone 100 meets the start condition 2, it instructs the computer 200 to perform multi-screen collaboration.

[0225] Not limited to the super terminal page, the embodiment of the present application can also trigger the mobile phone 100 / computer 200 to detect the startup conditions met by the device status of the mobile phone 100 in other ways, and automatically perform the collaborative mode corresponding to the startup conditions. No specific limitation is made here.

[0226] Interface Example 2

[0227] In the second embodiment of the interface, when the collaborative mode is initially started, the user can trigger the mobile phone 100 and computer 200 to enter different collaborative modes through different user operations. The above collaborative modes may include one or more of the following: input device sharing, screen projection, and application continuation. This embodiment of the application does not specifically limit the triggering method of the mobile phone 100 / computer 200 to trigger input device sharing, screen projection, and application continuation.

[0228] Exemplarily, FIG6A to FIG6G show that a user triggers two devices to enter different collaborative modes through different input operations.

[0229] In some embodiments, the mobile phone 100 receives a response to a preset operation 2 of the user, instructing the computer 200 to share the input device with the mobile phone 100 .

[0230] In some embodiments, the computer 200 receives a response to the user's preset operation 2, and the computer 200 shares the input device with the mobile phone 100. For example, FIG6A and FIG6B illustrate an interface implementation in which the user shares the input device with the mobile phone 100 through the computer 200.

[0231] As shown in Figure 6A , a user opens the input device sharing settings page 31 on a computer 200. The settings page 31 displays a list of available terminal devices and a list of available input devices. The user can select one or more terminal devices from the terminal device list as target devices, and select one or more input devices from the input device list to share with the target devices. As shown in Figure 6A , the terminal device list may include one or more device columns for available terminal devices, such as a device column 201 for a mobile phone 100. The device column 201 for a mobile phone 100 may include a selection control 201A, a device identifier 201B for the mobile phone 100, a connection status 201C, and a control control 201D. The connection status 201C indicates the current connection status between the mobile phone 100 and the computer 200. The connection status can be one of: disconnected, connected, and unpaired. The control 201D is used to disconnect when the connection status is connected, trigger a connection when the connection status is disconnected, and trigger both pairing and connection when the connection status is unpaired. The input device list may include one or more device columns of sharable input devices, such as a device column 202 for a mouse. The device column 202 for a mouse may include a device identifier 202A for the mouse and a selection control 202B.

[0232] As shown in Figure 6B, the user selects the mobile phone 100 by selecting the control 201A and selects the mouse by selecting the control 202B. The setting page 31 also includes the completion control 203; the computer 200 receives and responds to the user's input operation on the completion control 203, and shares the mouse with the mobile phone 100.

[0233] In some embodiments, the mobile phone 100 receives a preset operation 3 in response to the user and projects the preset screen display content to the computer 200 .

[0234] In some embodiments, the computer 200 receives a predetermined operation 3 in response to the user, instructing the mobile phone 100 to project the predetermined screen display content to the computer 200. The predetermined operation 3 is different from the predetermined operation 2. For example, Figures 6C to 6E illustrate an interface implementation in which a user triggers the mobile phone 100 to project the screen of the mobile phone 100 to the computer 200.

[0235] As shown in Figure 6C, the user brings the NFC area of ​​the mobile phone 100 close to the NFC area of ​​the computer 200 to trigger the NFC connection between the mobile phone 100 and the computer 200, thereby enabling the desktop of the mobile phone 100 to be projected to the computer 200. In one implementation, after the user brings the NFC area of ​​the mobile phone 100 close to the NFC area of ​​the computer 200, the mobile phone 100 can detect the NFC signal of the computer 200 and display the prompt box 204 shown in Figure 6D on the current display interface. The prompt box 204 includes a prompt message 204A, a connection control 204B, and a cancel control 204C. The prompt message 204A is used to prompt the user to click the connection control 204B to trigger the screen projection to the computer 200; the cancel control 204C is used to close the prompt box 204.

[0236] As shown in Figures 6D and 6E, the mobile phone 100 receives and responds to the input operation (such as touch operation) for the connection control 204B, and sends the projection data to the computer 200; the computer 200 displays the projection window 15 corresponding to the mobile phone 100 on the current display interface (such as the main interface 11 of the computer 200) based on the above projection data, and the projection window 15 displays the main interface 13 from the mobile phone 100.

[0237] In some embodiments, the mobile phone 100 receives a preset operation 4 in response to the user, instructing the computer 200 to perform an application connection on the application 2 currently running on the mobile phone 100 .

[0238] In some embodiments, computer 200 receives a user's preset action 4 and, in response to the user's request, performs an application connection to application 2 currently running on mobile phone 100. Preset action 4 is different from preset action 3 and preset action 2. For example, Figures 6F to 6G illustrate an interface implementation in which a user triggers computer 200 to connect application 2 on mobile phone 100 via computer 200.

[0239] In an embodiment of the present application, when a user opens a new application, the computer 200 can switch the application running in the foreground to the background, add the application window of the running application to the multi-task queue as a historical task, and store a snapshot of the user interface last displayed in the above application window as an interface snapshot of the historical task.

[0240] FIG6F illustrates a historical task interface 32 on computer 200. This interface includes an icon 205 for computer 200 and an icon for a terminal device with which computer 200 has established a collaborative connection, such as an icon 206 for mobile phone 100. The interface 32 can be used to display both the historical tasks of computer 200 and the historical tasks of mobile phone 100. As shown in FIG6F , the icon 205 for computer 200 is selected. At this point, the interface 32 displays task cards for the historical tasks of each application in the multitasking queue of computer 200. As shown in FIG6F and FIG6G , upon receiving and responding to an input operation directed to the icon 206 for mobile phone 100, computer 200 switches the icon 206 for mobile phone 100 from an unselected state to a selected state and displays task cards for the historical tasks of each application in the multitasking queue of mobile phone 100 on the interface 32, such as a task card 207 for an instant messaging application. Task card 207 displays a snapshot of the chat page 16 of contact "Henry."

[0241] In some embodiments, as shown in Figures 6G and 6H, upon receiving and responding to an input operation on the task card 207 of the instant messaging application, the computer 200 performs an application connection on the instant messaging application and displays the chat page 19 of the contact "Henry" corresponding to the task card 207 in the application connection window 17.

[0242] Interface Example 3

[0243] In the third embodiment of the interface, the mobile phone 100 and the computer 200 can automatically switch the collaborative mode according to the device status of the mobile phone 100. Specifically, when the mobile phone 100 and the computer 200 have currently started the first collaborative mode, if the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the startup conditions corresponding to the second collaborative mode, the mobile phone 100 / computer 200 automatically switches the first collaborative mode to the second collaborative mode, that is, stops the first collaborative mode and starts the second collaborative mode. For example, the first collaborative mode is input device sharing, and the second collaborative mode is multi-screen collaboration (such as screen projection, application continuation); or, the first collaborative mode is multi-screen collaboration, and the second collaborative mode is input device sharing.

[0244] In some embodiments, the computer 200 currently shares the input device with the mobile phone 100. If the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the start condition 2, the input device sharing is automatically switched to screen projection, that is, the computer 200 stops sharing the input device with the mobile phone 100, and the mobile phone 100 projects the screen to the computer 200.

[0245] In some embodiments, the mobile phone 100 is currently projecting the screen to the computer 200. If the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the start condition 1, the screen projection will be automatically switched to input device sharing, that is, the mobile phone 100 stops projecting the screen to the computer 200, and the computer 200 shares the input device with the mobile phone 100.

[0246] In some embodiments, the computer 200 currently shares the input device with the mobile phone 100. If the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the start condition 2, the input device sharing is automatically switched to application continuation, that is, the computer 200 stops sharing the input device with the mobile phone 100, and performs application continuation on the application 2 currently running on the mobile phone 100.

[0247] In some embodiments, the computer 200 performs an application connection on the application 2 running on the mobile phone 100. If the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the start condition 1, the application connection is automatically switched to input device sharing, that is, the computer 200 closes the application connection window, and the computer 200 shares the input device with the mobile phone 100.

[0248] In some embodiments, the computer 200 currently shares the input device with the mobile phone 100. If the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the start condition 2, the input device sharing is switched to multi-screen collaboration (such as screen casting, application continuation). In some embodiments, the mobile phone 100 and the computer 200 are currently performing multi-screen collaboration. If the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the start condition 1, the multi-screen collaboration is switched to input device sharing. Among them, the above-mentioned switching of input device sharing to multi-screen collaboration includes: if the page running in the foreground of the terminal device 100 belongs to an application that cannot be applied to continue, then multi-screen collaboration refers to the mobile phone 100 casting the preset screen display content to the computer 200; otherwise, multi-screen collaboration refers to the computer 200 performing application continuation on the application running in the foreground of the mobile phone 100.

[0249] For the above-mentioned starting condition 1 and starting condition 2, please refer to the relevant description of interface embodiment 1.

[0250] In one implementation, launch condition 1 includes any of the following: the phone 100 is not in a false touch prevention scenario, the false touch prevention function is not enabled on the phone 100, the screen of the phone 100 is facing the user, the phone 100 is tilted upright, and the distance between the phone 100 and the computer 200 is less than distance 2. Launch condition 2 includes: the device status of the phone 100 does not meet launch condition 1, and the page running in the foreground of the phone 100 is an application that cannot be continued. Launch condition 3 includes: the device status of the phone 100 does not meet launch condition 1, and the page running in the foreground of the phone 100 is an application that can be continued.

[0251] For example, as shown in (a) of Figure 7, the mobile phone 100 is in an upright posture, or / and, the mobile phone 100 is facing the user, the device status of the mobile phone 100 meets the start condition 1, and the computer 200 currently shares the input device (such as a mouse) with the mobile phone 100.

[0252] As shown in FIG7(b), when the mobile phone 100 detects that the mobile phone 100 is placed horizontally and / or is not facing the user, and an application that cannot be connected to the mobile phone 100 is running in the foreground (e.g., the main interface 13), the mobile phone 100 stops the input events of the input device of the computer 200, instructs the computer 200 to stop sharing the mouse with the mobile phone 100, and casts the currently displayed user interface (e.g., the main interface 13) to the computer 200, and the computer 200 displays the user interface from the mobile phone 100 in the casting window 15. Conversely, after FIG7(b), if the user switches the mobile phone 100 from a horizontal position back to an upright position, and / or switches the screen from not facing the user back to facing the user, the mobile phone 100 detects that the device state of the mobile phone 100 meets the start condition 1, and can stop casting and instruct the computer 200 to switch casting back to input device sharing.

[0253] As shown in FIG7(c), mobile phone 100 displays a chat page 16 for contact "Henry" in an instant messaging application. When mobile phone 100 detects that mobile phone 100 is placed horizontally and / or not facing the user, and that an instant messaging application capable of application continuation is running in the foreground, mobile phone 100 instructs computer 200 to stop sharing input devices with mobile phone 100 and instructs computer 200 to perform application continuation of mobile phone 100's instant messaging application, displaying a chat page 19 for contact "Henry" in the instant messaging application in application continuation window 17. Conversely, after FIG7(c), if the user switches mobile phone 100 from a horizontal position back to an upright position, and / or switches the screen from not facing the user back to facing the user, mobile phone 100 detects that the device state of mobile phone 100 meets start condition 1 and can instruct computer 200 to switch application continuation back to input device sharing.

[0254] For example, as shown in (a) of FIG8 , the distance between the mobile phone 100 and the computer 200 is less than distance 2, the current device state of the mobile phone 100 satisfies the start condition 1, and the computer 200 currently shares the input device (eg, a mouse) with the mobile phone 100 .

[0255] As shown in FIG8(b), the user moves the mobile phone 100 away from the computer 200, and the mobile phone 100 detects that the distance between the mobile phone 100 and the computer 200 is greater than or equal to distance 2, and the mobile phone 100 is running a system application that cannot be connected to the foreground, the mobile phone 100 instructs the computer 200 to stop sharing the input device with the mobile phone 100, and casts the currently displayed main interface 13 to the computer 200, and the computer 200 displays the main interface 13 from the mobile phone 100 in the casting window 15. On the contrary, after FIG8(b), if the user moves the mobile phone 100 closer to the computer 200 so that the distance between the mobile phone 100 and the computer 200 is less than distance 2, then when the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the start condition 1, it can instruct the computer 200 to switch the casting back to input device sharing.

[0256] As shown in FIG8(c), the user moves mobile phone 100 away from computer 200, and mobile phone 100 displays a chat page 16 of the contact "Henry" in the instant messaging application. When mobile phone 100 detects that the distance between mobile phone 100 and computer 200 is greater than or equal to distance 2, and that an instant messaging application capable of application connection is running in the foreground of mobile phone 100, mobile phone 100 instructs computer 200 to stop sharing the input device with mobile phone 100 and instructs computer 200 to connect the mobile phone 100's instant messaging application to the application connection window 17, displaying a chat page 19 of the contact "Henry" in the instant messaging application. Conversely, after FIG8(c), if the user moves mobile phone 100 closer to computer 200 so that the distance between mobile phone 100 and computer 200 is less than distance 2, mobile phone 100 / computer 200 detects that the device status of mobile phone 100 meets start condition 1 and can instruct computer 200 to switch application connection back to input device sharing.

[0257] Interface Example 4

[0258] In the fourth embodiment of the interface, the user can manually switch the collaborative mode. Specifically, when the mobile phone 100 and the computer 200 are in the first collaborative mode, the user can trigger the mobile phone 100 and the computer 200 to switch the first collaborative mode to the second collaborative mode by acting on the preset operation of the mobile phone 100 / computer 200, that is, stop the first collaborative mode and start the second collaborative mode. For example, the first collaborative mode is input device sharing, and the second collaborative mode is multi-screen collaboration (such as screen projection, application continuation); or, the first collaborative mode is multi-screen collaboration, and the second collaborative mode is input device sharing.

[0259] In some embodiments, the computer 200 is currently sharing an input device with the mobile phone 100. The mobile phone 100 / computer 200 receives and responds to the user's preset operation 5 to control the computer 200 and the mobile phone 100 to switch from input device sharing to screen projection. In some embodiments, the mobile phone 100 is currently projecting its screen to the computer 200. The mobile phone 100 / computer 200 receives and responds to the user's preset operation 6 to control the computer 200 and the mobile phone 100 to switch from screen projection to input device sharing.

[0260] In some embodiments, the computer 200 currently shares its input device with the mobile phone 100. The mobile phone 100 / computer 200 receives and responds to the user's preset operation 5. When the page running in the foreground of the mobile phone 100 is an application that cannot be continued, the computer 200 and the mobile phone 100 are controlled to switch the input device sharing to screen casting. When the page running in the foreground of the mobile phone 100 is an application that can be continued, the computer 200 and the mobile phone 100 are controlled to switch the input device sharing to application continuation. In some embodiments, the mobile phone 100 currently casts its screen to the computer 200. The mobile phone 100 / computer 200 receives and responds to the user's preset operation 6. The computer 200 and the mobile phone 100 are controlled to switch the screen casting to input device sharing. In some embodiments, the computer 200 currently performs application continuation on the application 2 of the mobile phone 100. The mobile phone 100 / computer 200 receives and responds to the user's preset operation 7. The computer 200 and the mobile phone 100 are controlled to switch the application continuation to input device sharing.

[0261] The embodiment of the present application does not specifically limit preset operation 5, preset operation 6, and preset operation 7.

[0262] Exemplarily, FIG. 9A to FIG. 9D show an interface implementation for manually switching the collaborative mode.

[0263] In an embodiment of the present application, after the computer 200 shares the input device with the mobile phone 100, the user can use the input device of the computer 200 (such as a mouse) to drag the pictures, videos, folders, files (such as word documents), user interfaces corresponding to application windows, and other content displayed on the mobile phone 100 to the computer 200 for display and operation.

[0264] In some embodiments, as shown in Figure 9A, after the computer 200 shares the input device with the mobile phone 100, the user can use the input device of the computer 200 (such as a mouse) to drag the user interface currently displayed by the mobile phone 100 (such as the main interface 13) to the screen of the computer 200, thereby triggering the computer 200 and the mobile phone 100 to switch the input device sharing to screen projection.

[0265] In one implementation, as shown in FIG9A , after the cursor corresponding to the mouse moves to the screen of the mobile phone 100, the mobile phone 100 detects that the user selects the main interface 13 with the mouse (for example, by long pressing the left button of the mouse on a blank area of ​​the main interface 13), and obtains and displays the interface snapshot 301 of the main interface 13; then, when it is detected that the user moves the mouse, the cursor and the interface snapshot 301 are controlled to move with the mouse, and the relative positions of the cursor and the interface snapshot 301 remain unchanged, presenting a visual effect of dragging the main interface 13; as the mouse continues to move, if the cursor moves After moving to the right border of the mobile phone 100 screen and continuing to move toward the outside of the screen, the cursor dragging interface snapshot 301 penetrates the left border of the screen of the computer 200; the computer 200 detects that the cursor dragging interface snapshot 301 has reached the screen of the computer 200, and the user releases the mouse (i.e. stops pressing the left button of the mouse), stops sharing the mouse with the mobile phone 100, and instructs the mobile phone 100 to cast the screen to the computer 200; the mobile phone casts the currently displayed main interface 13 to the computer 200, and the computer 200 displays the main interface 13 from the mobile phone 100 in the projection window 15.

[0266] In another implementation, the computer 200 switches the input device sharing to screen casting after detecting that the cursor dragging interface snapshot 301 moves onto the screen of the computer 200. In another implementation, the mobile phone 100 instructs the computer 200 to switch the input device sharing to screen casting after detecting that the cursor dragging interface snapshot 301 moves off the screen of the mobile phone 100.

[0267] In an embodiment of the present application, the mobile phone 100 may send the interface snapshot 301 to the computer 200 after obtaining the interface snapshot 301, or may send the interface snapshot 301 to the computer 200 when the cursor crosses the screen boundary of the mobile phone 100, which is not specifically limited here.

[0268] In some embodiments, referring to FIG9A , after the computer 200 shares an input device (e.g., a mouse) with the mobile phone 100, the user can use the mouse to drag the user interface (e.g., the main interface 13) currently displayed on the mobile phone 100 to the screen of the computer 200. If the dragged user interface is determined to be an application that cannot be applied for continuation, the computer 200 switches the input device sharing to screen casting. In one implementation, along with the interface snapshot 301, the mobile phone 100 also sends an identifier 1 to the computer 200, which is used to indicate that the main interface 13 corresponding to the interface snapshot 301 is not an application that can be applied for continuation.

[0269] In some embodiments, when the cursor drags the interface snapshot 301, the interface snapshot 301 moves out of the area of ​​the mobile phone screen and is moved into the screen of the computer 200. That is, while the computer 200 displays the first area of ​​the interface snapshot 301 that moves into the computer screen, the mobile phone 100 displays the second area outside the first area of ​​the interface snapshot 301.

[0270] For example, as shown in FIG9A , the cursor is spaced a first distance from the right border of the interface snapshot, and a second distance from the left border of the interface snapshot. In one implementation, when the cursor is dragging interface snapshot 301 across the mobile phone screen and the distance between the cursor and the right border of the mobile phone screen decreases below the first value and continues to decrease, the first area of ​​interface snapshot 301 moves off the mobile phone screen and onto the computer screen. While mobile phone 100 displays the first area of ​​interface snapshot 301, computer 200 displays a second area of ​​interface snapshot 301 outside the first area. The first and second areas are divided by the right border of the mobile phone screen, with the second area being the left half of interface snapshot 301. As the cursor drags interface snapshot 301 onto the computer screen, the greater the distance between the cursor and the left border of the computer screen, the larger the first area of ​​the computer screen moves into. When the distance between the cursor and the left border of the computer screen decreases below the second value, computer 200 displays the first area of ​​interface snapshot 301 while mobile phone 100 displays the second area of ​​interface snapshot 301. Similarly, in subsequent embodiments, when the cursor drags a window, interface snapshot, or other object to move, the computer 200 and the mobile phone 100 may respectively display a portion of the dragged object, which will not be described in detail in the subsequent embodiments.

[0271] In some embodiments, as shown in FIG9A , while the cursor is dragging interface snapshot 301 across the mobile phone screen, computer 200 does not display interface snapshot 301; while the cursor is dragging interface snapshot 301 onto the computer screen, mobile phone 100 stops displaying interface snapshot 301. Similarly, in subsequent embodiments, while the cursor is dragging an object such as a window or interface snapshot, computer 200 and mobile phone 100 cannot simultaneously display part of the dragged object, and subsequent embodiments will not be described in detail.

[0272] As shown in Figure 9B, after the computer 200 displays the projection window 15 of the mobile phone 100, the user can use the mouse to drag the projection window 15 back to the screen of the mobile phone 100, thereby triggering the mobile phone 100 and the computer 200 to switch the projection to input device sharing.

[0273] In one implementation, as shown in FIG9B , the projection window 15 displays the chat page 16; the computer 200 detects that the user selects the projection window 15 with the mouse (for example, by long pressing the left button of the mouse on the blank area of ​​the projection window 15); when it detects that the user moves the mouse, the cursor and the projection window 15 are controlled to move with the mouse, and the relative positions of the cursor and the projection window 15 remain unchanged, presenting a visual effect of the cursor dragging the projection window 15 to move; as the mouse continues to move, the computer 200 detects that the cursor moves to the left border of the computer 200 screen, moves to the left border of the computer 200 screen and stays at the left border for longer than a preset value, or moves When the cursor reaches the left edge of the computer 200 screen and continues to move toward the outside of the screen, an interface snapshot 302 is sent to the mobile phone 100, instructing the mobile phone 100 to stop projecting the screen to the computer 200 and share the input device (such as a mouse) with the mobile phone 100; after sharing the mouse with the mobile phone 100, as the mouse continues to move, the mobile phone 100 displays the cursor dragging interface snapshot 302 penetrating the right edge of the screen of the mobile phone 100; after the mobile phone 100 detects that the cursor drags the interface snapshot 302 to the screen of the mobile phone 100 and the user releases the mouse, it stops displaying the interface snapshot 302 of the projection window 15, runs in the foreground and displays the chat page 16 in the projection window 15. Referring to Figure 9B, when the user drags the projection window 15 of the computer 200 back to the screen of the mobile phone 100, the user interface displayed in the projection window 15 (such as the main interface 13) and the user interface displayed on the mobile phone 100 (such as the chat page 16) can be different or the same.

[0274] Among them, the computer 200 can obtain the interface snapshot 302 of the projection window 15 when it detects that the user selects the projection window 15, or it can obtain the interface snapshot 302 when it detects that the cursor moves to the left boundary of the computer 200 screen and continues to move to the outside of the screen. No specific limitation is made here.

[0275] In some embodiments, referring to Figure 9C, after the computer 200 shares the input device with the mobile phone 100, the user can use the input device of the computer 200 (such as a mouse) to drag the user interface currently displayed on the mobile phone 100 (such as the chat page 16) to the screen of the computer 200, thereby triggering the computer 200 to switch the input device sharing to application connection.

[0276] In one implementation, as shown in Figure 9C, when the mobile phone 100 detects that the user selects the chat page 16 with the mouse, it obtains and displays the interface snapshot 303 of the chat page 16; when it detects that the user moves the mouse, it controls the cursor and the interface snapshot 303 to move with the mouse; if the cursor moves to the right boundary of the mobile phone 100 screen and continues to move to the outside of the screen, the computer 200 displays the cursor dragging the interface snapshot 303 to penetrate the left boundary of the screen of the computer 200; after the computer 200 detects that the cursor drags the interface snapshot 303 to the screen of the computer 200 and the user releases the mouse, it stops sharing the mouse with the mobile phone 100, and performs application connection on the chat page 16 of the instant messaging application of the mobile phone 100, and displays the application connection window 17.

[0277] In one implementation, along with the interface snapshot 303, the mobile phone 100 may also send an identifier 2 to the computer 200, where the identifier 2 is used to indicate that the chat page 16 corresponding to the interface snapshot 303 is an application that can be continued. The interface snapshot 303 can refer to the description of the interface snapshot 301, which will not be repeated here.

[0278] In some embodiments, as shown in FIG9D , after the computer 200 displays the application connection window 17 of the instant messaging application, the user can use the mouse to drag the application connection window 17 to the screen of the mobile phone 100 , thereby triggering the computer 200 to switch the application connection to input device sharing.

[0279] In one implementation, as shown in FIG9D , the computer 200 displays the application continuation window 17; when the computer 200 detects that the user selects the application continuation window 17 with the mouse and moves the mouse, the computer 200 controls the cursor and the application continuation window 17 to move with the mouse; if the cursor moves to the left border of the computer 200 screen, moves to the left border of the computer 200 screen and stays at the left border for longer than a preset value, or moves to the left border of the computer 200 screen and continues to move toward the outside of the screen, the computer 200 closes the application continuation window 17 in the background and shares the input device (such as the mouse) with the mobile phone 100; after the mouse is shared with the mobile phone 100, as the mouse continues to move, the mobile phone 100 displays an interface snapshot 304 of the application continuation window 17 dragged by the cursor and penetrating the right border of the screen of the mobile phone 100; as shown in FIG9D , the mobile phone 100 detects that the cursor drags the interface snapshot 304 to the screen of the mobile phone 100, and after the user releases the mouse, the interface snapshot 304 of the application continuation window 17 stops being displayed. In one implementation, the application connection window 17 displays a chat page 19 of the contact "Henry" of the computer version of the instant messaging application. After the mobile phone 100 stops displaying the interface snapshot 304 of the application connection window 17, the chat page 16 of "Henry" of the mobile version of the instant messaging application runs in the foreground.

[0280] As shown in FIG9D , when the application connection window 17 of the computer 200 is dragged back to the screen of the mobile phone 100 , the application displayed in the application connection window 17 (eg, an instant messaging application) and the application displayed on the mobile phone 100 (eg, a desktop application) may be different or the same.

[0281] For example, with respect to the manual switching collaborative mode method described in Figures 9A to 9D , Figure 9E illustrates a software implementation process for manually switching collaborative mode. As shown in Figures 3 and 9E , the UI framework can include system controls; the basic capability subsystem set of the system service layer can also include a drag framework.

[0282] In some embodiments, as shown in FIG9F , when the computer 200 shares the input device with the mobile phone 100, the user can use the input device (such as a mouse or a touch screen pad) to drag the interface snapshot of the user interface 1 (such as the main interface 13 or the chat page 16) currently displayed on the mobile phone 100 to the screen of the computer 200, thereby triggering the computer 200 to switch the input device sharing to multi-screen collaboration (such as screen casting or application continuation). For example, referring to FIG9E , FIG9A and FIG9C , after the computer 200 shares the input device with the mobile phone 100, when the mouse cursor moves to the screen of the mobile phone 100, the multi-mode input subsystem of the mobile phone 100 can monitor the input event of the input device (such as a mouse) shared by the computer 200 for the mobile phone 100; for example, when the user selects the user interface currently displayed on the mobile phone 100 by long pressing the left mouse button, the multi-mode input subsystem monitors the mouse input event 1 and sends the input event 1 to the system control of the UI framework; the system control is based on the interface layout of the user interface running in the foreground , it is determined that input event 1 is used to select user interface 1, the system control instructs the drag framework to obtain an interface snapshot of user interface 1, and the drag framework can share the interface snapshot with the computer 200 through the distributed data management module; the mobile phone 100 displays the mouse cursor and the above interface snapshot; when the user drags the above interface snapshot with the mouse, the multi-mode input subsystem detects the mouse input event 2 and sends the input event 2 to the traversal service; the traversal service determines the mouse cursor position according to input event 2, and instructs the drag framework to display the cursor dragging interface snapshot according to the above cursor position; the traversal service detects When the cursor moves to the boundary of the mobile phone 100 and continues to move to the outside of the screen, the drag frame is instructed to stop displaying the cursor drag interface snapshot and send a crossing information to the crossing service of the computer 200; the crossing service of the computer 200 informs the multi-mode input subsystem and the drag frame of the above crossing information; the above crossing information instructs the multi-mode input subsystem of the computer 200 to locally process the mouse input event; the drag frame of the computer 200 obtains the interface snapshot of the user interface 1 sent by the drag frame of the mobile phone 100, triggering the computer 200 to display the cursor and the interface snapshot; the user drags the interface snapshot by the mouse When moving on the screen of computer 200, the multi-mode input subsystem detects the mouse input event 4 and sends the input event 4 to the crossing service; the crossing service determines the cursor position based on input event 4, and instructs the drag framework to display the cursor dragging interface snapshot based on the above cursor position; when the user stops dragging the interface snapshot with the mouse and releases the mouse, the multi-mode input subsystem detects the mouse input event 5 and sends the input event 5 to the system control of the UI framework. The system control of the UI framework triggers the mobile phone 100 to switch the input device sharing to multi-screen collaboration (such as screen projection, application continuation) based on input event 5.

[0283] In some embodiments, as shown in Figure 9B or Figure 9D, when the computer 200 performs multi-screen collaboration (such as screen projection, application continuation) on the mobile phone 100, the user can use an input device (such as a mouse, a touch screen pad) to drag the multi-screen collaboration window (such as a screen projection window, an application continuation window) currently displayed on the computer 200 back to the screen of the mobile phone 100, thereby triggering the mobile phone 100 to switch the multi-screen collaboration back to input device sharing. For example, referring to Figures 9E, 9B and 9D, the computer 200 displays the multi-screen collaborative window of the mobile phone 100. After the user selects the multi-screen collaborative window by long pressing the left mouse button, as the mouse moves, the computer 200 displays the cursor dragging the multi-screen collaborative window on the screen; when the cursor moves to the boundary of the computer 200 and continues to move to the outside of the screen, the multi-modal input subsystem of the computer 200 detects the input event 6 of the mouse and sends the input event 6 to the crossing service; based on the input event 6, the crossing service instructs the drag framework to obtain an interface snapshot of the multi-screen collaborative window, triggering the computer 200 to switch the multi-screen collaboration to input device sharing, and send crossing information to the crossing service of the mobile phone 100, and the drag framework shares the interface snapshot with the mobile phone 100. After completing the sharing of the input device, the mobile phone 100's traversal service notifies the multi-mode input subsystem of the above traversal information, instructing the mobile phone 100's multi-mode input subsystem to monitor the mouse input event; the mobile phone 100's multi-mode input subsystem detects input event 7 and sends input event 7 to the traversal service. The traversal service instructs the drag framework to obtain the interface snapshot of the multi-screen collaborative window shared by the computer 200. The drag framework triggers the mobile phone 100 to display the cursor dragging the interface snapshot; when the user stops dragging the interface snapshot with the mouse and releases the mouse, the mobile phone 100's multi-mode input subsystem detects mouse input event 8 and sends input event 8 to the system control of the UI framework. Based on input event 8, the system control triggers the mobile phone 100 to stop displaying the interface snapshot of the multi-screen collaborative window. In some embodiments, based on input event 8, the system control also triggers the mobile phone 100 to continue running the user interface in the multi-screen collaborative window.

[0284] Exemplarily, FIG. 9F to FIG. 9I show another interface implementation of collaborative mode switching.

[0285] In some embodiments, the mobile phone 100 and / or the computer 200 are provided with switch controls for each collaborative mode of the connected collaborative device, such as a switch control for input device sharing, a switch control for screen projection, and a switch control for application connection; a switch control for a collaborative mode is used to control the start / stop of the collaborative mode.

[0286] In some embodiments, at most one collaborative mode switch control can be in an on state, and the user can control the switching of collaborative modes through the switch control.

[0287] For example, as shown in FIG9F , the computer 200 is provided with switch controls for each collaborative mode of the connected mobile phone 100. After the computer 200 establishes a collaborative connection with the mobile phone 100 through the hyperterminal, the status bar of the computer 200 displays an icon 401 of the connected mobile phone 100; upon receiving and responding to an input operation (e.g., a click operation) on the icon 401 of the mobile phone 100, the computer 200 displays the settings page 21 of the mobile phone 100, which displays a control 402, a switch control 403 for input device sharing, and a switch control 404 for multi-screen collaboration.

[0288] Among them, the control 402 is used to disconnect the collaborative connection with the mobile phone 100. When the switch control 403 for input device sharing is switched to the on state (i.e., "ON"), the computer 200 shares the input device with the mobile phone 100; when the switch control 403 for input device sharing is switched to the off state (i.e., "OFF"), the computer 200 stops sharing the input device with the mobile phone 100.

[0289] In one implementation, when the switch control 404 of multi-screen collaboration is switched to the on state (i.e., “ON”), the computer 200 instructs the mobile phone 100 to project the screen to the computer 200; when the switch control 404 of multi-screen collaboration is switched to the off state (i.e., “OFF”), the computer 200 instructs the mobile phone 100 to stop projecting the screen to the computer 200.

[0290] In another implementation, when the switch control 404 of multi-screen collaboration is switched to the on state (i.e., "ON"), the computer 200 instructs the mobile phone 100 to perform multi-screen collaboration. If the page running in the foreground of the mobile phone 100 is an application that cannot be applied and continued, the mobile phone 100 casts the screen to the computer 200. If the page running in the foreground of the mobile phone 100 is an application that can be applied and continued, the computer 200 performs application continuation on the page running in the foreground of the mobile phone 100; when the switch control 404 of multi-screen collaboration is switched to the off state (i.e., "OFF"), the computer 200 and the mobile phone 100 stop multi-screen collaboration (i.e., stop screen casting / application continuation).

[0291] In some embodiments, at most one collaborative mode switch control of the input device sharing switch control 403 and the multi-screen collaboration switch control 404 can be in the on state, that is, input device sharing and multi-screen collaboration (i.e., screen projection or application continuation) cannot be turned on at the same time, and the switch control can be used to switch the collaborative mode.

[0292] For example, as shown in Figure 9F, when the computer 200 shares the input device with the mobile phone 100, the switch control 403 for input device sharing is in the on state, and the switch control 404 for multi-screen collaboration is in the off state; upon receiving and responding to an input operation (such as a click operation) for the switch control 404 for multi-screen collaboration, the computer 200 switches the switch control 403 for input device sharing to the off state, stops sharing the input device with the mobile phone 100, switches the switch control 403 for multi-screen collaboration to the on state, and instructs the mobile phone 100 to perform multi-screen collaboration; when the page running in the foreground of the mobile phone 100 is the main interface 13 that cannot be applied, the mobile phone 100 casts the screen to the computer 200, and the computer 200 displays the projection window 15, in which the main interface 13 of the mobile phone 100 is displayed.

[0293] For example, as shown in Figure 9G, the mobile phone 100 is projected to the computer 200. When the computer 200 displays the projection window 15, the switch control 403 for input device sharing is in the off state, and the switch control 404 for multi-screen collaboration is in the on state; upon receiving and responding to an input operation (such as a click operation) on the switch control 403 for input device sharing, the computer 200 switches the switch control 403 for multi-screen collaboration to the off state, instructing the mobile phone 100 to stop projecting the screen, and switches the switch control 403 for input device sharing to the on state, sharing the input device with the mobile phone 100.

[0294] For example, as shown in Figure 9H, when the computer 200 shares the input device with the mobile phone 100, the switch control 403 for input device sharing is in the on state, and the switch control 404 for multi-screen collaboration is in the off state; upon receiving and responding to an input operation (such as a click operation) for the switch control 404 for multi-screen collaboration, the computer 200 switches the switch control 403 for input device sharing to the off state, stops sharing the input device with the mobile phone 100, switches the switch control 403 for multi-screen collaboration to the on state, and instructs the mobile phone 100 to perform multi-screen collaboration; when the page running in the foreground of the mobile phone 100 is a chat page 16 that can be applied for continuation, the computer 200 performs application continuation on the chat page 16 of the instant messaging application and displays the application continuation window 17 of the instant messaging application.

[0295] For example, as shown in Figure 9I, when the computer 200 displays the application connection window 17, the switch control 403 shared by the input device is in the off state, and the switch control 404 for multi-screen collaboration is in the on state; upon receiving and responding to an input operation (such as a click operation) on the switch control 403 shared by the input device, the computer 200 switches the switch control 403 for multi-screen collaboration to the off state, closes the application connection window 17, and switches the switch control 403 shared by the input device to the on state, sharing the input device with the mobile phone 100.

[0296] In the embodiments of the present application, the switch controls for each collaborative mode are not limited to the settings page 21, and can also be set on other pages of the computer 200, without specific limitation here. In the examples provided in Figures 9G to 9I, screen casting and application continuation can share a switch control; in other examples, screen casting and application continuation can also correspond to a switch control respectively, without specific limitation here.

[0297] In other embodiments, when the mobile phone 100 and the computer 200 are in the first collaborative mode, if the mobile phone 100 / computer 200 detects that the mobile phone 100 meets the starting conditions corresponding to the second collaborative mode, the mobile phone 100 and the computer 200 are automatically triggered to start the second collaborative mode, that is, the first collaborative mode and the second collaborative mode can be performed at the same time. In one implementation, the starting conditions corresponding to the first collaborative mode and the starting conditions corresponding to the second collaborative mode can coexist, and the developer can define the starting conditions of each collaborative mode according to the usage requirements of the actual scenario. The embodiment of the present application does not specifically limit the starting conditions of each collaborative mode. For example, the first collaborative mode is input device sharing, and the second collaborative mode is multi-screen collaboration; or, the first collaborative mode is multi-screen collaboration, and the second collaborative mode is input device sharing.

[0298] In some embodiments, the computer 200 currently shares the input device with the mobile phone 100. If it is detected that the device status of the mobile phone 100 meets the start conditions for screen projection, the computer 200 will instruct the mobile phone 100 to project the screen to the computer 200 while sharing the input device with the mobile phone 100.

[0299] In some embodiments, the computer 200 currently shares the input device with the mobile phone 100. If it is detected that the device status of the mobile phone 100 meets the startup conditions for application continuation, the computer 200 will share the input device with the mobile phone 100 while performing application continuation on the application 2 running in the foreground of the mobile phone 100.

[0300] In some embodiments, the mobile phone 100 is currently projecting its screen to the computer 200. If it is detected that the device status of the mobile phone 100 meets the start-up conditions for sharing the input device, the computer 200 will share the input device with the mobile phone 100 while displaying the projection window corresponding to the mobile phone 100.

[0301] In some embodiments, when the computer 200 continues the application 1 of the mobile phone 100, if it detects that the device status of the mobile phone 100 meets the start conditions for sharing the input device, the computer 200 will share the input device with the mobile phone 100 while continuing the application 2 of the mobile phone 100.

[0302] In other embodiments, when the mobile phone 100 and the computer 200 are in the first collaborative mode, the user can manually trigger the mobile phone 100 and the computer 200 to start the second collaborative mode, that is, the first collaborative mode and the second collaborative mode can be carried out simultaneously. Specifically, when the mobile phone 100 and the computer 200 are in the first collaborative mode, the user can trigger the mobile phone 100 and the computer 200 to start the second collaborative mode by acting on the preset operation (such as the preset operation 5, preset operation 6 and preset operation 7) on the mobile phone 100 / computer 200.

[0303] The present application provides a multi-device collaboration method, which is applied to a multi-device collaboration system. The multi-device collaboration system includes a first terminal and a second terminal, where the second terminal is connected to a first input device. The method includes:

[0304] S101: A first terminal establishes a communication connection with a second terminal; the first terminal displays a first user interface.

[0305] In the embodiment of the present application, the first terminal may be the aforementioned terminal device 100, and the second terminal may be the aforementioned terminal device 200.

[0306] S102: The second terminal shares at least one input device with the first terminal, the first terminal and the second terminal are in an input device sharing mode, and the at least one input device includes the first input device;

[0307] S103: In response to a first operation on the first input device, the first terminal displays a first interface snapshot of the first user interface being dragged by a cursor of the first input device to move;

[0308] S104. After the cursor drags the snapshot of the first interface to the display screen of the second terminal, the second terminal and the first terminal switch the input device sharing mode to the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal, and the multi-screen collaboration window displays content associated with the first user interface.

[0309] In the embodiment of the present application, the first input device may be an input device such as a mouse, a stylus pen, or a touchpad, and no specific limitation is given here.

[0310] In one implementation, the first user interface is the user interface of the first application, and the multi-screen collaborative window displays the user interface of the first application. The method further includes: the second terminal detecting a second operation acting on the multi-screen collaborative window; and in response to the second operation, the second terminal displaying the user interface of the second application in the multi-screen collaborative window. For example, with reference to the relevant description of FIG9C , the first application can be the aforementioned instant messaging application, the first user interface can be the aforementioned chat page 16, the first interface snapshot can be the aforementioned interface snapshot 303, the multi-screen collaborative window can be the aforementioned application continuation window 17, and the first operation can be the preset operation 5, for example, the first operation includes long-pressing the left mouse button to select the chat page 16 and then moving the mouse.

[0311] In one implementation, the first user interface is a system interface. For example, referring to the relevant description of FIG9A , the first user interface may be the aforementioned main interface 13, the first interface snapshot may be the aforementioned interface snapshot 301, the multi-screen collaboration window may be the aforementioned projection window 15, and the first operation may be the preset operation 5, for example, the first operation includes long-pressing the left mouse button to select the main interface 13 and then moving the mouse.

[0312] In some embodiments, the cursor is at a first value from a first boundary of the first interface snapshot, and at a second value from a second boundary of the first interface snapshot; the first boundary is parallel to the second boundary, and the first boundary and the first boundary are respectively one of a left boundary, a right boundary, an upper boundary, and a lower boundary; the above method further includes: when the cursor drags the first interface snapshot on the screen of the first terminal, if the distance between the cursor and the first boundary of the screen of the first terminal is less than the first value and continues to decrease, the first area of ​​the first interface snapshot moves out of the screen of the first terminal and into the screen of the second terminal; while the second terminal displays the first area of ​​the first interface snapshot, the first terminal displays the area outside the first area of ​​the first interface snapshot; when the cursor drags the first interface snapshot on the screen of the second terminal, if the distance between the cursor and the second boundary of the screen of the second terminal increases, the first area moved into the screen of the second terminal increases; when the distance between the cursor and the second boundary of the screen of the second terminal is less than the second value, while the second terminal displays the first area of ​​the first interface snapshot, the first terminal displays the area outside the first area of ​​the first interface snapshot. For example, with reference to the relevant description of FIG. 9A , the first boundary may be the right boundary, and the second boundary may be the left boundary.

[0313] In one implementation, the distance between the cursor and the boundary of the first interface snapshot is a first value, and the method further includes: when the cursor drags the first interface snapshot to move on the screen of the first terminal, when the distance between the cursor and the boundary of the screen of the first terminal is less than the first value and continues to decrease, the first area of ​​the first interface snapshot moves out of the screen of the first terminal and into the screen of the second terminal.

[0314] In one implementation, after the cursor drags the first interface snapshot to the screen of the second terminal, the first terminal displays the first area in the first interface snapshot, and the second terminal displays the second area in the first interface snapshot except the first area; alternatively, after the cursor drags the first interface snapshot to the screen of the second terminal, the first terminal does not display the first interface snapshot, and the second terminal displays a partial area in the first interface snapshot.

[0315] In one implementation, when the first user interface belongs to an application that cannot be continued by the second terminal, the multi-screen collaborative mode is a projection mode; the above-mentioned multi-screen collaborative window displays content associated with the first user interface, including: the multi-screen collaborative window displays the first user interface; when the first user interface belongs to the first application that can be continued by the second terminal, the multi-screen collaborative mode is an application continuation mode; the first user interface displayed by the first terminal is the user interface of the first version of the first application; the multi-screen collaborative window displays content associated with the first user interface, including: the multi-screen collaborative window displays the user interface corresponding to the first user interface in the second version of the first application; the first version is an application version applicable to the device type of the first terminal, and the second version is an application version applicable to the device type of the second terminal. In the embodiment of the present application, the first version can be the aforementioned version 1 (for example, the mobile phone version), and the second version can be the aforementioned version 2 (for example, the computer version).

[0316] In one implementation, the multi-screen collaboration mode is a projection mode; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays the first user interface. For example, referring to the relevant description of FIG9A , the mobile phone 100 displays the main interface 11, and the projection window 15 displays the main interface 11.

[0317] In one implementation, the multi-screen collaboration mode is an application continuation mode; the first user interface displayed on the first terminal is the user interface of the first version of the first application; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays a user interface corresponding to the first user interface in the second version of the first application; the first version is an application version suitable for the device type of the first terminal, and the second version is an application version suitable for the device type of the second terminal. For example, referring to the relevant description of FIG9A , the mobile phone 100 displays a chat page 16 of the mobile version of the instant messaging application "Henry", and the application continuation window 17 displays a chat page 19 of the computer version of the instant messaging application "Henry".

[0318] In one implementation, the first user interface belongs to a first application that can be applied to a connection, and when the second terminal does not have the first application installed, before switching the input device sharing mode to the multi-screen collaboration mode, the method includes: the second terminal displays a prompt message, the prompt message is used to prompt the user to install the first application on the second terminal; in response to a third operation, the second terminal installs the first application and determines that the multi-screen collaboration mode is the application connection mode; in response to a fourth operation, the multi-screen collaboration mode is determined to be the projection mode. Exemplarily, with reference to the relevant description of FIG4I, the prompt message can be the aforementioned prompt message 113; the third operation is used to install the first application (for example, clicking the installation control 114); the fourth operation can be used to cancel the installation of the first application (for example, clicking the cancel control 115).

[0319] In one implementation, the method also includes: when the first terminal and the second terminal are in a multi-screen collaborative mode, in response to the fifth operation of the first input device, the second terminal displays the cursor of the first input device and drags the multi-screen collaborative window to move; during the process of dragging the multi-screen collaborative window with the cursor, after the cursor moves to the boundary of the screen of the second terminal, the second terminal and the first terminal switch the multi-screen collaborative mode to the input device sharing mode, the second terminal stops displaying the multi-screen collaborative window, and shares the above-mentioned at least one input device with the first terminal.

[0320] In one implementation, the multi-screen collaboration window displays a second user interface; when the multi-screen collaboration mode is switched to the input device sharing mode, the method further includes: the first terminal displays content associated with the second user interface.

[0321] In one implementation, the multi-screen collaboration mode is a projection mode, and the first terminal displays content associated with the second user interface, including: the first terminal displays the second user interface. Exemplarily, with reference to the relevant description of FIG9B , the second user interface may be the aforementioned chat page 16, the multi-screen collaboration window may be the aforementioned projection window 15, and the fifth operation includes long-pressing the left mouse button to select the projection window 15 and then moving the mouse.

[0322] In one implementation, the multi-screen collaboration mode is an application continuation mode, the second user interface displayed in the multi-screen collaboration window is the user interface of the second version of the first application, and the first terminal displays content associated with the second user interface, including: the first terminal displays the user interface corresponding to the second user interface in the first version of the first application; the first version is the application version applicable to the device type of the first terminal; and the second version is the application version applicable to the device type of the second terminal. For example, referring to the relevant description of FIG9D , the first application can be the aforementioned instant messaging application, the second user interface can be the chat page 19 of the aforementioned contact "Henry", and the multi-screen collaboration window can be the aforementioned application continuation window 17.

[0323] In one implementation, the method further includes: when the first terminal and the second terminal are not in the input device sharing mode and the multi-screen collaborative mode, the first terminal or the second terminal receives and responds to the sixth operation, and when the device status of the first terminal meets the first start-up condition, the first terminal and the second terminal are triggered to start the input device sharing mode, and the second terminal shares the above-mentioned at least one input device with the first terminal. Exemplarily, with reference to the relevant description of Figures 4C and 4D, according to the device status of the mobile phone 100, you can choose to start the input device sharing mode. When the device status of the first terminal meets the second start-up condition, the first terminal and the second terminal are triggered to start the multi-screen collaborative mode, and the second terminal displays the multi-screen collaborative window corresponding to the first terminal. Exemplarily, with reference to the relevant description of Figures 4E to 4H, according to the device status of the mobile phone 100, you can choose to start the multi-screen collaborative mode.

[0324] In the embodiment of the present application, the sixth operation may be the aforementioned preset operation 1, the first starting condition may be the aforementioned starting condition 1, and the second starting condition may be the aforementioned starting condition 2.

[0325] In one implementation, the method further includes: when the first terminal and the second terminal are in multi-screen collaborative mode, when it is detected that the device status of the first terminal meets the first starting condition, triggering the first terminal and the second terminal to switch the multi-screen collaborative mode to the input device sharing mode. When the first terminal and the second terminal are in input device sharing mode, when it is detected that the device status of the first terminal meets the second starting condition, triggering the first terminal and the second terminal to switch the input device sharing mode to the multi-screen collaborative mode. Exemplarily, with reference to the relevant description of Figures 7 and 8, the collaborative mode can be automatically switched according to the device status of the mobile phone 100.

[0326] In one implementation, the first terminal and the second terminal switch the input device sharing mode to the multi-screen collaboration mode, including: when the second terminal cannot apply the continuation to the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the screen projection mode, and the second terminal displays the screen projection content from the first terminal in the multi-screen collaboration window; when the second terminal can apply the continuation to the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the application continuation mode, and the second terminal performs the application continuation to the application running in the foreground of the first terminal in the multi-screen collaboration window. Exemplarily, with reference to the relevant descriptions of Figures 7 and 8, when automatically switching to the multi-screen collaboration mode, it can be automatically determined whether the multi-screen collaboration mode is the screen projection mode or the application continuation mode based on the page running in the foreground.

[0327] In one implementation, the first activation condition includes one or more of the following: not being in a false touch prevention scenario, the false touch prevention function is not enabled, the screen is facing the user, the posture is upright, and the distance from the second terminal is less than a third value; the second activation condition includes: the device status of the first terminal does not meet the first activation condition. The third value can be the aforementioned distance 2.

[0328] The various implementation modes of this application can be combined arbitrarily to achieve different technical effects.

[0329] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0330] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0331] In short, the above description is only an embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made based on the disclosure of this application should be included in the scope of protection of this application.

Claims

1. A multi-device collaboration method, characterized in that: Applied to a multi-device collaborative system, the multi-device collaborative system includes a first terminal and a second terminal, the second terminal is connected to a first input device, and the method includes: The first terminal establishes a communication connection with the second terminal; The first terminal displays a first user interface; The second terminal shares at least one input device with the first terminal, the first terminal and the second terminal are in an input device sharing mode, and the at least one input device includes the first input device; In response to a first operation of the first input device, the first terminal displays a first interface snapshot of the first user interface being dragged by a cursor of the first input device to move; After the cursor drags the first interface snapshot to the display screen of the second terminal, the second terminal and the first terminal switch the input device sharing mode to the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal, and the multi-screen collaboration window displays content associated with the first user interface.

2. The method according to claim 1, characterized in that The first user interface is a user interface of a first application, and the multi-screen collaboration window displays the user interface of the first application. The method further includes: The second terminal detects a second operation acting on the multi-screen collaborative window; In response to the second operation, the second terminal displays a user interface of a second application in the multi-screen collaborative window.

3. The method according to claim 1, characterized in that The first user interface is a system interface.

4. The method according to claim 1, characterized in that The distance between the cursor and the boundary of the first interface snapshot is a first value, and the method further includes: When the cursor drags the first interface snapshot on the screen of the first terminal, when the distance between the cursor and the boundary of the screen of the first terminal is less than the first value and continues to decrease, the first area of ​​the first interface snapshot moves out of the screen of the first terminal and moves into the screen of the second terminal.

5. The method according to claim 1, characterized in that After the cursor drags the first interface snapshot to the screen of the second terminal, the first terminal displays the first area in the first interface snapshot, and the second terminal displays the second area in the first interface snapshot except the first area, or, After the cursor drags the first interface snapshot to the screen of the second terminal, the first terminal does not display the first interface snapshot, and the second terminal displays a partial area of ​​the first interface snapshot.

6. The method according to claim 1, characterized in that When the first user interface belongs to an application that the second terminal cannot apply to continue, the multi-screen collaboration mode is a projection mode; and the multi-screen collaboration window displays content associated with the first user interface, including: The multi-screen collaborative window displays the first user interface; When the first user interface belongs to the first application that can be applied and continued by the second terminal, the multi-screen collaboration mode is an application continuation mode; the first user interface displayed by the first terminal is a user interface of the first version of the first application; and the multi-screen collaboration window displays content associated with the first user interface, including: The multi-screen collaborative window displays a user interface corresponding to the first user interface in the second version of the first application; the first version is an application version suitable for the device type of the first terminal, and the second version is an application version suitable for the device type of the second terminal.

7. The method according to claim 1, characterized in that The first user interface belongs to a first application that can be applied continuously, and when the second terminal does not install the first application, before switching the input device sharing mode to the multi-screen collaboration mode, the method includes: The second terminal displays prompt information, where the prompt information is used to prompt the user to install the first application on the second terminal; In response to the third operation, the second terminal installs the first application and determines that the multi-screen collaboration mode is an application continuation mode; In response to the fourth operation, it is determined that the multi-screen collaborative mode is a screen projection mode.

8. The method according to claim 1, characterized in that The method further comprises: When the first terminal and the second terminal are in the multi-screen collaborative mode, in response to a fifth operation of the first input device, the second terminal displays the cursor of the first input device dragging the multi-screen collaborative window to move; During the process of dragging the multi-screen collaborative window with the cursor, after the cursor moves to the boundary of the screen of the second terminal, the second terminal and the first terminal switch the multi-screen collaborative mode to the input device sharing mode, the second terminal stops displaying the multi-screen collaborative window, and shares the at least one input device with the first terminal.

9. The method according to claim 8, characterized in that The multi-screen collaborative window displays a second user interface; When the multi-screen collaboration mode is switched to the input device sharing mode, the method further includes: The first terminal displays content associated with the second user interface.

10. The method according to claim 9, characterized in that The multi-screen collaboration mode is a screen projection mode, and the first terminal displays content associated with the second user interface, including: The first terminal displays the second user interface.

11. The method according to claim 9, characterized in that The multi-screen collaboration mode is an application continuation mode, the second user interface displayed in the multi-screen collaboration window is a user interface of a second version of the first application, and the first terminal displays content associated with the second user interface, including: The first terminal displays a user interface in the first version of the first application corresponding to the second user interface; the first version is an application version suitable for the device type of the first terminal; the second version is an application version suitable for the device type of the second terminal.

12. The method according to any one of claims 1 to 11, characterized in that: The method further comprises: When the first terminal and the second terminal are not in the input device sharing mode and the multi-screen collaboration mode, the first terminal or the second terminal receives and responds to a sixth operation, and when the device state of the first terminal meets a first starting condition, triggers the first terminal and the second terminal to start the input device sharing mode, and the second terminal shares the at least one input device with the first terminal; When the device state of the first terminal meets the second starting condition, the first terminal and the second terminal are triggered to start the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal.

13. The method according to any one of claims 1 to 12, characterized in that: The method further comprises: When the first terminal and the second terminal are in the multi-screen collaborative mode, when it is detected that the device state of the first terminal meets the first starting condition, triggering the first terminal and the second terminal to switch the multi-screen collaborative mode to the input device sharing mode; When the first terminal and the second terminal are in the input device sharing mode, when it is detected that the device state of the first terminal meets the second starting condition, the first terminal and the second terminal are triggered to switch the input device sharing mode to the multi-screen collaboration mode.

14. The method according to claim 12 or 13, characterized in that The first terminal and the second terminal switch the input device sharing mode to the multi-screen collaboration mode, including: When the second terminal cannot connect to the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the screen projection mode, and the second terminal displays the screen projection content from the first terminal in the multi-screen collaborative window; When the second terminal can perform application continuation on the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the application continuation mode, and the second terminal performs application continuation on the application running in the foreground of the first terminal in the multi-screen collaborative window.

15. The method according to any one of claims 12 to 14, characterized in that The first start condition includes one or more of the following: not in the accidental touch prevention scene, the accidental touch prevention function is not turned on, the screen faces the user, the posture tends to be upright, and the distance from the second terminal is less than a third value; The second start condition includes: a device state of the first terminal does not satisfy the first start condition.

16. A multi-device collaboration method, characterized in that: The method comprises: The first terminal establishes a communication connection with the second terminal, and the second terminal is connected to a first input device; The first terminal displays a first user interface; The first terminal and the second terminal are in an input device sharing mode, the first terminal shares at least one input device of the second terminal, and the at least one input device includes the first input device; In response to a first operation of the first input device, the first terminal displays a first interface snapshot of the first user interface being dragged by a cursor of the first input device to move; After the cursor drags the first interface snapshot out of the screen of the first terminal, the first terminal switches the input device sharing mode with the second terminal to the multi-screen collaboration mode, and sends collaboration data of the multi-screen collaboration window to the second terminal, the collaboration data is used to determine the display content of the multi-screen collaboration window, and the multi-screen collaboration window is used to be displayed on the screen of the second terminal. The multi-screen collaboration window is used to display content associated with the first user interface.

17. The method according to claim 16, characterized in that The first user interface is a user interface of a first application, the multi-screen collaboration window is used to display content in the first application that is associated with the first user interface, and the multi-screen collaboration window is also used to display a user interface of a second application.

18. The method according to claim 16, characterized in that The first user interface is a system interface.

19. The method according to claim 16, characterized in that When the first user interface belongs to an application that the second terminal cannot apply to continue, the multi-screen collaboration mode is a projection mode; and the content associated with the first user interface is the first user interface; When the first user interface belongs to the first application that can be applied and continued by the second terminal, the multi-screen collaboration mode is an application continuation mode; the first user interface displayed by the first terminal is a user interface of the first version of the first application; The content associated with the first user interface includes a user interface corresponding to the first user interface in the second version of the first application; The first version is an application version applicable to a device type of the first terminal, and the second version is an application version applicable to a device type of the second terminal.

20. The method according to claim 16, characterized in that The method further comprises: When the first terminal and the second terminal are in the multi-screen collaborative mode, in response to the first information sent by the second terminal, the first terminal switches the multi-screen collaborative mode with the second terminal to the input device sharing mode; The first information is sent after the cursor drags the multi-screen collaborative window to move on the screen of the second terminal and the cursor moves to the boundary of the screen of the second terminal.

21. The method according to claim 20, characterized in that The multi-screen collaborative window displays a second user interface; when the multi-screen collaborative mode with the second terminal is switched to the input device sharing mode, the method further includes: The first terminal displays content associated with the second user interface.

22. The method according to claim 21, characterized in that The multi-screen collaboration mode is a screen projection mode, and the first terminal displays content associated with the second user interface, including: The first terminal displays the second user interface.

23. The method according to claim 21, characterized in that The multi-screen collaboration mode is an application continuation mode, the second user interface displayed in the multi-screen collaboration window is a user interface of a second version of the first application, and the first terminal displays content associated with the second user interface, including: The first terminal displays a user interface in the first version of the first application corresponding to the second user interface; the first version is an application version suitable for the device type of the first terminal; the second version is an application version suitable for the device type of the second terminal.

24. A multi-device collaboration method, characterized in that: The method comprises: The second terminal establishes a communication connection with the first terminal, and the second terminal is connected to a first input device; The second terminal shares at least one input device with the first terminal, the first terminal and the second terminal are in an input device sharing mode, and the at least one input device includes the first input device; After the cursor of the first input device drags the first interface snapshot of the first user interface displayed by the first terminal to the display screen of the second terminal, the second terminal switches the input device sharing mode with the first terminal to the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal, and the multi-screen collaboration window displays content associated with the first user interface.

25. The method according to claim 24, characterized in that The first user interface is a user interface of a first application, and the multi-screen collaboration window displays the user interface of the first application. The method further includes: The second terminal detects a second operation acting on the multi-screen collaborative window; In response to the second operation, the second terminal displays a user interface of a second application in the multi-screen collaborative window.

26. The method according to claim 24, characterized in that The first user interface is a system interface.

27. The method according to claim 24, characterized in that When the first user interface belongs to an application that the second terminal cannot apply to continue, the multi-screen collaboration mode is a projection mode; and the multi-screen collaboration window displays content associated with the first user interface, including: The multi-screen collaborative window displays the first user interface; When the first user interface belongs to the first application that can be applied and continued by the second terminal, the multi-screen collaboration mode is an application continuation mode; the first user interface displayed by the first terminal is a user interface of the first version of the first application; and the multi-screen collaboration window displays content associated with the first user interface, including: The multi-screen collaborative window displays a user interface corresponding to the first user interface in the second version of the first application; the first version is an application version suitable for the device type of the first terminal, and the second version is an application version suitable for the device type of the second terminal.

28. The method according to claim 24, characterized in that The first user interface belongs to a first application that can be applied continuously, and when the second terminal does not install the first application, before switching the input device sharing mode to the multi-screen collaboration mode, the method includes: The second terminal displays prompt information, where the prompt information is used to prompt the user to install the first application on the second terminal; In response to the third operation, the second terminal installs the first application and determines that the multi-screen collaboration mode is an application continuation mode; In response to the fourth operation, it is determined that the multi-screen collaborative mode is a screen projection mode.

29. The method according to claim 24, characterized in that The method further comprises: When the first terminal and the second terminal are in the multi-screen collaborative mode, in response to a fifth operation of the first input device, the second terminal displays the cursor of the first input device dragging the multi-screen collaborative window to move; During the process of dragging the multi-screen collaborative window with the cursor, after the cursor moves to the boundary of the screen of the second terminal, the second terminal and the first terminal switch the multi-screen collaborative mode to the input device sharing mode, the second terminal stops displaying the multi-screen collaborative window, and shares the at least one input device with the first terminal.

30. A multi-device collaboration method, characterized in that: Applied to a multi-device collaborative system, the multi-device collaborative system includes a first terminal and a second terminal, and the method includes: When the first terminal and the second terminal are not in the input device sharing mode and the multi-screen collaboration mode, the first terminal or the second terminal receives and responds to a sixth operation, and when the device state of the first terminal meets a first starting condition, triggers the first terminal and the second terminal to start the input device sharing mode, and the second terminal shares the at least one input device with the first terminal; When the device state of the first terminal meets the second starting condition, the first terminal and the second terminal are triggered to start the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal.

31. The method according to claim 30, characterized in that The method further comprises: When the first terminal and the second terminal are in the multi-screen collaborative mode, when it is detected that the device state of the first terminal meets the first starting condition, triggering the first terminal and the second terminal to switch the multi-screen collaborative mode to the input device sharing mode; When the first terminal and the second terminal are in the input device sharing mode, when it is detected that the device state of the first terminal meets the second starting condition, the first terminal and the second terminal are triggered to switch the input device sharing mode to the multi-screen collaboration mode.

32. The method according to claim 30 or 31, characterized in that The first terminal and the second terminal switch the input device sharing mode to the multi-screen collaboration mode, including: When the second terminal cannot connect to the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the screen projection mode, and the second terminal displays the screen projection content from the first terminal in the multi-screen collaborative window; When the second terminal can perform application continuation on the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the application continuation mode, and the second terminal performs application continuation on the application running in the foreground of the first terminal in the multi-screen collaborative window.

33. The method according to any one of claims 30 to 31, characterized in that The first start condition includes one or more of the following: not in the accidental touch prevention scene, the accidental touch prevention function is not turned on, the screen faces the user, the posture tends to be upright, and the distance from the second terminal is less than a third value; The second start condition includes: a device state of the first terminal does not satisfy the first start condition.

34. A terminal device, characterized in that: include: A processor and a memory, wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor reads the computer instructions from the memory, the terminal device executes the multi-device collaboration method as described in any one of claims 16 to 23 or 24 to 29.

35. A computer-readable storage medium, characterized in that: It includes computer instructions, which, when executed on a terminal, enable the terminal device to execute a multi-device collaboration method as described in any one of claims 16 to 23 or 24 to 29.

Images