Multi-device cooperation method and related device

Abstract

The invention discloses a multi-device cooperation method and a related device, and the method comprises the steps: a first terminal and a second terminal establish communication connection, and the second terminal is connected with a first input device; the first terminal displays a first user interface; the second terminal shares at least one input device with the first terminal, and the first terminal and the second terminal are in an input device sharing mode; in response to a first operation of the first input device, the first terminal displays that a cursor of the first input device drags a first interface snapshot of the first user interface to move; after the cursor drags the first interface snapshot to move to a display screen of the second terminal, the second terminal and the first terminal switch an input device sharing mode into a multi-screen cooperation mode, the second terminal displays a multi-screen cooperation window corresponding to the first terminal, and the multi-screen cooperation window displays content associated with the first user interface. In this way, the real-time demand cooperation mode of the user can be matched, complex operation is not needed, and the user experience is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to a multi-device collaboration method and related apparatus. Background Art

[0002] With the continuous development of terminal technology, users have more and more types and functions of electronic devices. Limited by their own hardware structure and system performance, different types of devices usually have different capabilities and advantages. Multiple devices of users, combined with the performance advantages of each device, conduct collaborative control of device interconnection and distributed services, bringing users a better user experience. After multiple devices are collaboratively connected, multiple functional modes can be collaboratively realized between multiple devices. For example, mobile terminals such as mobile phones can migrate multimedia content (such as pictures, audio, and video, etc.) to smart screens for continued playback and projection; laptops can share the mouse with tablets, and users can control tablets through the mouse of laptops, and so on.

[0003] After establishing a communication connection between devices, it remains to be studied what functional mode to choose to collaboratively implement in order to match the user's real-time needs as much as possible. Summary of the invention

[0004] The embodiment of the present application provides a multi-device collaboration method that can appropriately select a collaboration mode to match the user's real-time needs as much as possible without the need for complicated operations, thereby effectively improving the user experience.

[0005] In a first aspect, the present application provides a multi-device collaboration method, which is applied to a multi-device collaboration system, wherein the multi-device collaboration system includes a first terminal and a second terminal, and the second terminal is connected to a first input device, and the method includes: the first terminal establishes a communication connection with the second terminal; the first terminal displays a first user interface; the second terminal shares at least one input device with the first terminal, and the first terminal and the second terminal are in an input device sharing mode, and the at least one input device includes a first input device; in response to a first operation of the first input device, the first terminal displays a first interface snapshot of the first user interface dragged by a cursor of the first input device to move; after the cursor drags the first interface snapshot to the display screen of the second terminal, the second terminal and the first terminal switch the input device sharing mode to a multi-screen collaboration mode, and the second terminal displays a multi-screen collaboration window corresponding to the first terminal, and the multi-screen collaboration window displays content associated with the first user interface.

[0006] In the implementation of the embodiment of the present application, in the input device sharing mode, dragging the user interface displayed by the first terminal to the screen of the second terminal through the input device of the second terminal can trigger the first terminal and the second terminal to switch the input device sharing mode to the multi-screen collaboration mode. The multi-screen collaboration window displayed by the second terminal can be used to control the first terminal without the need to control the first terminal through the input device of the second terminal. In this way, according to the user's drag operation, the multi-screen collaboration mode can be quickly selected and switched to, which matches the user's real-time demand for the collaboration mode between devices and effectively improves the user experience.

[0007] In one implementation, the first user interface is a user interface of a first application, and the multi-screen collaborative window displays the user interface of the first application, and the method further includes: the second terminal detects a second operation acting on the multi-screen collaborative window; in response to the second operation, the second terminal displays the user interface of the second application in the multi-screen collaborative window. In implementing the embodiment of the present application, in the multi-screen collaborative mode, the user can manipulate the multi-screen collaborative window of the second terminal to display the user interfaces of different applications of the first terminal.

[0008] In one implementation, the first user interface is a system interface. When implementing the embodiment of the present application, in the multi-screen collaboration mode, the multi-screen collaboration window of the second terminal can display the system interface of the first terminal.

[0009] In one implementation, the distance between the cursor and the boundary of the first interface snapshot is a first value, and the method further includes: when the cursor drags the first interface snapshot to move on the screen of the first terminal, when the distance between the cursor and the boundary of the screen of the first terminal is less than the first value and continues to decrease, the first area of ​​the first interface snapshot moves out of the screen of the first terminal and moves into the screen of the second terminal. In implementing the embodiment of the present application, when the cursor drags the interface snapshot to move, a partial area of ​​the interface snapshot moves out of the screen of the first terminal and moves into the screen of the second terminal. In this way, the visual effect is smoother and more beautiful.

[0010] In one implementation, after the cursor drags the first interface snapshot to the screen of the second terminal, the first terminal displays the first area in the first interface snapshot, and the second terminal displays the second area in the first interface snapshot except the first area, or, after the cursor drags the first interface snapshot to the screen of the second terminal, the first terminal does not display the first interface snapshot, and the second terminal displays a partial area in the first interface snapshot. In the implementation of the embodiment of the present application, during the process of dragging the interface snapshot with the cursor, when the cursor moves to the second terminal, while the second terminal displays a partial area of ​​the interface snapshot, the first terminal can display the remaining area of ​​the interface snapshot or stop displaying the interface snapshot, thereby enriching the visual effect when dragging the interface snapshot.

[0011] In one implementation, when the first user interface belongs to an application that cannot be applied and continued by the second terminal, the multi-screen collaboration mode is a projection mode; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays the first user interface; when the first user interface belongs to the first application that can be applied and continued by the second terminal, the multi-screen collaboration mode is an application continuation mode; the first user interface displayed by the first terminal is the user interface of the first version of the first application; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays the user interface corresponding to the first user interface in the second version of the first application; the first version is an application version applicable to the device type of the first terminal, and the second version is an application version applicable to the device type of the second terminal. In implementing the embodiments of the present application, without user selection, it is possible to adaptively determine whether the multi-screen collaboration mode is specifically a projection mode or an application continuation mode based on the page running in the foreground.

[0012] In one implementation, the multi-screen collaboration mode is a projection mode; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays the first user interface. When the embodiment of the present application is implemented, when switching to the projection mode, the user interface currently displayed on the first terminal can be projected to the multi-screen collaboration window of the second terminal.

[0013] In one implementation, the multi-screen collaboration mode is an application continuation mode; the first user interface displayed by the first terminal is the user interface of the first version of the first application; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays a user interface corresponding to the first user interface in the second version of the first application; the first version is an application version applicable to the device type of the first terminal, and the second version is an application version applicable to the device type of the second terminal. When implementing the embodiment of the present application, the first terminal runs the user interface of the first application in the foreground, and when switching to the application continuation mode, the second terminal opens the first application installed on the device in the multi-screen collaboration window, and then continues to display the first application.

[0014] In one implementation, the first user interface belongs to a first application that can be applied to continue, and when the second terminal has not installed the first application, before switching the input device sharing mode to the multi-screen collaboration mode, the method includes: the second terminal displays a prompt message, the prompt message is used to prompt the user to install the first application on the second terminal; in response to a third operation, the second terminal installs the first application, and determines that the multi-screen collaboration mode is the application continuation mode; in response to a fourth operation, determines that the multi-screen collaboration mode is the projection mode. When implementing the embodiment of the present application, the first terminal runs the first application in the foreground, and when switching to the application continuation mode, if the second terminal has not installed the first application, it can remind the user to install the first application, and switch to the application continuation mode after the installation is successful, otherwise it switches to the projection mode, thereby improving the user experience.

[0015] In one implementation, the method further includes: when the first terminal and the second terminal are in a multi-screen collaborative mode, in response to the fifth operation of the first input device, the second terminal displays the cursor of the first input device dragging the multi-screen collaborative window to move; during the process of dragging the multi-screen collaborative window by the cursor, after the cursor moves to the boundary of the screen of the second terminal, the second terminal and the first terminal switch the multi-screen collaborative mode to the input device sharing mode, the second terminal stops displaying the multi-screen collaborative window, and shares the above-mentioned at least one input device with the first terminal. In the implementation of the embodiment of the present application, in the multi-screen collaborative mode, the second terminal drags the multi-screen collaborative window to the screen boundary, which can trigger the multi-screen collaborative mode to switch to the input device sharing mode. Then, the first terminal can be directly controlled by the input device of the second terminal, without having to control the first terminal through the multi-screen collaborative window. In this way, according to the user's drag operation, the input device sharing mode is quickly selected and switched to, which matches the user's real-time demand for the collaborative mode between devices, and effectively improves the user experience.

[0016] In one implementation, the multi-screen collaboration window displays a second user interface; when the multi-screen collaboration mode is switched to the input device sharing mode, the method further includes: the first terminal displays content associated with the second user interface.

[0017] In one implementation, the multi-screen collaboration mode is a projection mode, and the first terminal displays content associated with the second user interface, including: the first terminal displays the second user interface. In this way, when the projection mode is switched to the input device sharing mode, the first terminal can also continue to display the projection content in the multi-screen collaboration window.

[0018] In one implementation, the multi-screen collaboration mode is an application continuation mode, the second user interface displayed in the multi-screen collaboration window is the user interface of the second version of the first application, and the first terminal displays content associated with the second user interface, including: the first terminal displays the user interface corresponding to the second user interface in the first version of the first application; the first version is an application version applicable to the device type of the first terminal; the second version is an application version applicable to the device type of the second terminal. In this way, when the application continuation mode is switched to the input device sharing mode, the first terminal can also continue to display the application in the multi-screen collaboration window.

[0019] In one implementation, the method further includes: when the first terminal and the second terminal are not in the input device sharing mode and the multi-screen collaboration mode, the first terminal or the second terminal receives and responds to the sixth operation, and when the device status of the first terminal meets the first start condition, the first terminal and the second terminal are triggered to start the input device sharing mode, and the second terminal shares the above-mentioned at least one input device with the first terminal; when the device status of the first terminal meets the second start condition, the first terminal and the second terminal are triggered to start the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal. In implementing the embodiment of the present application, when the collaboration mode is started, the collaboration mode that matches the device status of the first terminal is automatically selected from multiple collaboration modes to meet the real-time needs of the user as much as possible.

[0020] In one implementation, the method further includes: when the first terminal and the second terminal are in the multi-screen collaborative mode, when it is detected that the device state of the first terminal satisfies the first starting condition, triggering the first terminal and the second terminal to switch the multi-screen collaborative mode to the input device sharing mode; when the first terminal and the second terminal are in the input device sharing mode, when it is detected that the device state of the first terminal satisfies the second starting condition, triggering the first terminal and the second terminal to switch the input device sharing mode to the multi-screen collaborative mode. In implementing the embodiment of the present application, the device state of the first terminal can be detected, and the collaborative mode can be automatically switched to a collaborative mode that matches the device state.

[0021] In one implementation, the first terminal and the second terminal switch the input device sharing mode to the multi-screen collaboration mode, including: when the second terminal cannot apply continuity to the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the projection mode, and the second terminal displays the projection content from the first terminal in the multi-screen collaboration window; when the second terminal can apply continuity to the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the application continuation mode, and the second terminal performs application continuation on the application running in the foreground of the first terminal in the multi-screen collaboration window.

[0022] In one implementation, the first startup condition includes one or more of the following: not in the anti-mistouch scenario, the anti-mistouch function is not turned on, the screen is facing the user, the posture is inclined to be upright, and the distance from the second terminal is less than a third value; the second startup condition includes: the device status of the first terminal does not meet the first startup condition.

[0023] In the second aspect, the present application provides a multi-device collaboration method, characterized in that the method includes: a first terminal establishes a communication connection with a second terminal, and the second terminal is connected to a first input device; the first terminal displays a first user interface; the first terminal and the second terminal are in an input device sharing mode, and the first terminal shares at least one input device of the second terminal, and the at least one input device includes a first input device; in response to a first operation of the first input device, the first terminal displays a first interface snapshot of the first user interface dragged by a cursor of the first input device to move; after the cursor drags the first interface snapshot out of the screen of the first terminal, the first terminal switches the input device sharing mode with the second terminal to a multi-screen collaboration mode, and sends collaboration data of the multi-screen collaboration window to the second terminal, the collaboration data is used to determine the display content of the multi-screen collaboration window, and the multi-screen collaboration window is used to be displayed on the screen of the second terminal, and the multi-screen collaboration window is used to display content associated with the first user interface.

[0024] In the implementation of the embodiment of the present application, in the input device sharing mode, dragging the user interface displayed by the first terminal to the screen of the second terminal through the input device of the second terminal can trigger the first terminal and the second terminal to switch the input device sharing mode to the multi-screen collaboration mode. The multi-screen collaboration window displayed by the second terminal can be used to control the first terminal without the need to control the first terminal through the input device of the second terminal. In this way, according to the user's drag operation, the multi-screen collaboration mode can be quickly selected and switched to, which matches the user's real-time demand for the collaboration mode between devices and effectively improves the user experience.

[0025] In one implementation, the first user interface is a user interface of a first application, the multi-screen collaboration window is used to display content associated with the first user interface in the first application, and the multi-screen collaboration window is also used to display the user interface of a second application. In implementing the embodiment of the present application, in the multi-screen collaboration mode, a user can manipulate the multi-screen collaboration window of the second terminal to display the user interfaces of different applications of the first terminal.

[0026] In one implementation, the first user interface is a system interface. When implementing the embodiment of the present application, in the multi-screen collaboration mode, the multi-screen collaboration window of the second terminal can display the system interface of the first terminal.

[0027] In one implementation, when the first user interface belongs to an application that cannot be applied to the second terminal, the multi-screen collaboration mode is the projection mode; the content associated with the first user interface is the first user interface; when the first user interface belongs to the first application that can be applied to the second terminal, the multi-screen collaboration mode is the application continuation mode; the first user interface displayed by the first terminal is the user interface of the first version of the first application; the content associated with the first user interface includes the user interface corresponding to the first user interface in the second version of the first application; the first version is an application version applicable to the device type of the first terminal, and the second version is an application version applicable to the device type of the second terminal. When implementing the embodiments of the present application, without user selection, it is possible to adaptively determine whether the multi-screen collaboration mode is specifically the projection mode or the application continuation mode based on the page running in the foreground.

[0028] In one implementation, the method further includes: when the first terminal and the second terminal are in a multi-screen collaborative mode, in response to a first message sent by the second terminal, the first terminal switches the multi-screen collaborative mode with the second terminal to an input device sharing mode; the first message is sent after the cursor drags the multi-screen collaborative window on the screen of the second terminal and the cursor moves to the boundary of the screen of the second terminal. In implementing the embodiment of the present application, in the multi-screen collaborative mode, the second terminal drags the multi-screen collaborative window to the screen boundary, which can trigger the multi-screen collaborative mode to switch to the input device sharing mode. Then, the input device of the second terminal can be used to directly control the first terminal without having to control the first terminal through the multi-screen collaborative window. In this way, according to the user's drag operation, the input device sharing mode is quickly selected and switched to match the user's real-time demand for the collaborative mode between devices, effectively improving the user experience.

[0029] In one implementation, the multi-screen collaboration window displays a second user interface; when the multi-screen collaboration mode with the second terminal is switched to an input device sharing mode, the method further includes: the first terminal displays content associated with the second user interface.

[0030] In one implementation, the multi-screen collaboration mode is a projection mode, and the first terminal displays content associated with the second user interface, including: the first terminal displays the second user interface. In this way, when the projection mode is switched to the input device sharing mode, the first terminal can also continue to display the projection content in the multi-screen collaboration window.

[0031] In one implementation, the multi-screen collaboration mode is an application continuation mode, the second user interface displayed in the multi-screen collaboration window is the user interface of the second version of the first application, and the first terminal displays content associated with the second user interface, including: the first terminal displays the user interface corresponding to the second user interface in the first version of the first application; the first version is an application version applicable to the device type of the first terminal; the second version is an application version applicable to the device type of the second terminal. In this way, when the application continuation mode is switched to the input device sharing mode, the first terminal can also continue to display the application in the multi-screen collaboration window.

[0032] In the third aspect, the present application provides a multi-device collaboration method, the method comprising: a second terminal establishes a communication connection with a first terminal, and the second terminal is connected to a first input device; the second terminal shares the at least one input device with the first terminal, and the first terminal and the second terminal are in an input device sharing mode, and the at least one input device includes the first input device; after the cursor of the first input device drags a first interface snapshot of a first user interface displayed on the first terminal to a display screen of the second terminal, the second terminal switches the input device sharing mode with the first terminal to a multi-screen collaboration mode, and the second terminal displays a multi-screen collaboration window corresponding to the first terminal, and the multi-screen collaboration window displays content associated with the first user interface.

[0033] In the implementation of the embodiment of the present application, in the input device sharing mode, dragging the user interface displayed by the first terminal to the screen of the second terminal through the input device of the second terminal can trigger the first terminal and the second terminal to switch the input device sharing mode to the multi-screen collaboration mode. The multi-screen collaboration window displayed by the second terminal can be used to control the first terminal without the need to control the first terminal through the input device of the second terminal. In this way, according to the user's drag operation, the multi-screen collaboration mode can be quickly selected and switched to, which matches the user's real-time demand for the collaboration mode between devices and effectively improves the user experience.

[0034] In one implementation, the first user interface is a user interface of a first application, and the multi-screen collaborative window displays the user interface of the first application, and the method further includes: the second terminal detects a second operation acting on the multi-screen collaborative window; in response to the second operation, the second terminal displays the user interface of the second application in the multi-screen collaborative window. In implementing the embodiment of the present application, in the multi-screen collaborative mode, the user can manipulate the multi-screen collaborative window of the second terminal to display the user interfaces of different applications of the first terminal.

[0035] In one implementation, the first user interface is a system interface. When implementing the embodiment of the present application, in the multi-screen collaboration mode, the multi-screen collaboration window of the second terminal can display the system interface of the first terminal.

[0036] In one implementation, when the first user interface belongs to an application that cannot be applied to the second terminal, the multi-screen collaboration mode is a projection mode; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays the first user interface; when the first user interface belongs to a first application that can be applied to the second terminal, the multi-screen collaboration mode is an application continuation mode; the first user interface displayed by the first terminal is a user interface of the first version of the first application; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays a user interface corresponding to the first user interface in the second version of the first application; the first version is an application version applicable to the device type of the first terminal, and the second version is an application version applicable to the device type of the second terminal. In implementing the embodiments of the present application, without user selection, it is possible to adaptively determine whether the multi-screen collaboration mode is specifically a projection mode or an application continuation mode based on the page running in the foreground.

[0037] In one implementation, the first user interface belongs to a first application that can be applied to continue, and when the second terminal has not installed the first application, before switching the input device sharing mode to the multi-screen collaboration mode, the method includes: the second terminal displays a prompt message, the prompt message is used to prompt the user to install the first application on the second terminal; in response to a third operation, the second terminal installs the first application, and determines that the multi-screen collaboration mode is the application continuation mode; in response to a fourth operation, determines that the multi-screen collaboration mode is the projection mode. When implementing the embodiment of the present application, the first terminal runs the first application in the foreground, and when switching to the application continuation mode, if the second terminal has not installed the first application, it can remind the user to install the first application, and switch to the application continuation mode after the installation is successful, otherwise it switches to the projection mode, thereby improving the user experience.

[0038] In one implementation, the method further includes: when the first terminal and the second terminal are in a multi-screen collaborative mode, in response to the fifth operation of the first input device, the second terminal displays the cursor of the first input device dragging the multi-screen collaborative window to move; during the process of dragging the multi-screen collaborative window by the cursor, after the cursor moves to the boundary of the screen of the second terminal, the second terminal and the first terminal switch the multi-screen collaborative mode to the input device sharing mode, the second terminal stops displaying the multi-screen collaborative window, and shares the above-mentioned at least one input device with the first terminal. In the implementation of the embodiment of the present application, in the multi-screen collaborative mode, the second terminal drags the multi-screen collaborative window to the screen boundary, which can trigger the multi-screen collaborative mode to switch to the input device sharing mode. Then, the first terminal can be directly controlled by the input device of the second terminal, without having to control the first terminal through the multi-screen collaborative window. In this way, according to the user's drag operation, the input device sharing mode is quickly selected and switched to, which matches the user's real-time demand for the collaborative mode between devices, and effectively improves the user experience.

[0039] In a fourth aspect, the present application provides a multi-device collaboration method, which is applied to a multi-device collaboration system, wherein the multi-device collaboration system includes a first terminal and a second terminal, and the method includes: when the first terminal and the second terminal are not in an input device sharing mode and a multi-screen collaboration mode, the first terminal or the second terminal receives and responds to a sixth operation, and when the device status of the first terminal meets the first start-up condition, the first terminal and the second terminal are triggered to start the input device sharing mode, and the second terminal shares the above-mentioned at least one input device with the first terminal; when the device status of the first terminal meets the second start-up condition, the first terminal and the second terminal are triggered to start the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal. In implementing the embodiment of the present application, when the collaboration mode is started, the collaboration mode that matches the device status of the first terminal is automatically selected from a plurality of collaboration modes to meet the real-time needs of the user as much as possible.

[0040] In one implementation, the method further includes: when the first terminal and the second terminal are in the multi-screen collaborative mode, when it is detected that the device state of the first terminal satisfies the first starting condition, triggering the first terminal and the second terminal to switch the multi-screen collaborative mode to the input device sharing mode; when the first terminal and the second terminal are in the input device sharing mode, when it is detected that the device state of the first terminal satisfies the second starting condition, triggering the first terminal and the second terminal to switch the input device sharing mode to the multi-screen collaborative mode. In implementing the embodiment of the present application, the device state of the first terminal can be detected, and the collaborative mode can be automatically switched to a collaborative mode that matches the device state.

[0041] In one implementation, the first terminal and the second terminal switch the input device sharing mode to the multi-screen collaboration mode, including: when the second terminal cannot apply continuity to the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the projection mode, and the second terminal displays the projection content from the first terminal in the multi-screen collaboration window; when the second terminal can apply continuity to the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the application continuation mode, and the second terminal performs application continuation on the application running in the foreground of the first terminal in the multi-screen collaboration window.

[0042] In one implementation, the first startup condition includes one or more of the following: not in the anti-mistouch scenario, the anti-mistouch function is not turned on, the screen is facing the user, the posture is inclined to be upright, and the distance from the second terminal is less than a third value; the second startup condition includes: the device status of the first terminal does not meet the first startup condition.

[0043] In the fifth aspect, an embodiment of the present application provides a terminal device, comprising: a processor and a memory, wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code comprises computer instructions, and when the processor reads the computer instructions from the memory, the terminal device executes the multi-device collaboration method described in the first aspect.

[0044] In a sixth aspect, an embodiment of the present application provides a computer storage medium, including computer instructions. When the computer instructions are executed on a terminal device, the terminal device executes a multi-device collaboration method in any possible implementation of any of the above aspects.

[0045] In a seventh aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute a multi-device collaboration method in any possible implementation of any of the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of a system architecture of a communication system provided in an embodiment of the present application;

[0047] Figure 2A A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0048] Figure 2B A schematic diagram of a geographic coordinate system provided in an embodiment of the present application;

[0049] Figure 2C A schematic diagram of a device coordinate system provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of a software system architecture provided for an embodiment of the present application;

[0051] FIG. 4A to FIG. 4I A schematic diagram of an interface for automatically selecting a collaborative mode provided in an embodiment of the present application;

[0052] FIG. 5A to FIG. 5B A schematic diagram of the interface of a mobile phone hyperterminal page provided in an embodiment of the present application;

[0053] 6A to 6H A schematic diagram of an interface for manually selecting a collaborative mode provided in an embodiment of the present application;

[0054] Figure 7 and Figure 8 A schematic diagram of an interface for automatically switching collaborative modes provided in an embodiment of the present application;

[0055] 9A to 9D A schematic diagram of an interface for manually switching collaborative modes provided in an embodiment of the present application;

[0056] Fig.9E A software implementation process provided for an embodiment of the present application;

[0057] FIG. 9F to FIG. 9I A schematic diagram of the interface for manually switching collaborative modes provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0059] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more. The "upper", "lower", "left" and "right" involved in the embodiments of the present application are relative concepts and should not constitute a limitation on the embodiments of the present application.

[0060] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that the user can recognize. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icon, button, menu, tab, text box, dialog box, status bar, navigation bar, widget, etc. displayed on the display screen of an electronic device. The user interface can also be referred to as an interface or page.

[0061] After multiple terminal devices of a user (such as mobile phones and laptops) establish a communication connection, they can collaboratively implement various collaborative modes, such as input device sharing, multi-screen collaboration, etc. At present, the application scenarios of various collaborative modes are relatively limited. For example, multi-screen collaboration (such as screen projection, application continuation) is usually applied to mobile phones and large-screen devices (such as personal computers (PCs), TVs), and input device sharing is usually applied to large-screen devices and large-screen devices (such as PCs and tablet computers (PADs)); limited by communication resources (for example, input device sharing and screen projection both use WiFi P2P for communication), each terminal device usually only implements one of the collaborative modes of input device sharing and multi-screen collaboration at the same time, and the user needs to manually control the terminal device to turn on the required collaborative mode, and the operation of switching collaborative modes is relatively cumbersome. For example, to switch the input device sharing between the PAD and the PC to screen projection, the user needs to open the setting interface of input device sharing on the PC, and control the PC to turn off the input device sharing of the PAD through the setting interface; then, open the setting interface of screen projection on the PAD, and control the PAD to project the screen to the PC through the setting interface. The multi-device collaboration method provided in the embodiment of the present application is not limited to the application scenarios of the collaborative mode (for example, mobile phones and large-screen devices can perform multi-screen collaboration and share input devices). After the two terminal devices establish a communication connection, the collaborative mode can be appropriately selected to match the user's real-time needs as much as possible.

[0062] The following is an introduction to the communication system 10 involved in the multi-device collaboration method provided in an embodiment of the present application.

[0063] Figure 1 The system architecture of the communication system 10 provided in the embodiment of the present application is exemplarily shown. Figure 1 As shown, the communication system 10 includes at least two terminal devices, for example, a terminal device 100 and a terminal device 200 .

[0064] Among them, the terminal device 100 and the terminal device 200 have a display screen, and the terminal device 100 and the terminal device 200 can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, and can also be a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device (smart bracelet), a vehicle-mounted device, a smart home device (smart TV, smart screen, large screen device, etc.) and / or a smart city device. The device types of the terminal device 100 and the terminal device 200 can be the same or different; the embodiment of the present application does not impose any special restrictions on the specific types of the terminal device 100 and the terminal device 200. For example, the terminal device 100 is a mobile phone and the terminal device 200 is a laptop computer; or, the terminal device 100 is a tablet computer and the terminal device 200 is a mobile phone; or, the terminal device 100 is a tablet computer and the terminal device 200 is a laptop computer.

[0065] In an embodiment of the present application, the terminal device 100 and the terminal device 200 can establish a communication connection; based on the communication connection and multi-device collaboration technology, the terminal device 100 and the terminal device 200 can collaborate across systems and devices to achieve resource sharing and collaborative operation, and the above communication connection can also be referred to as a collaborative connection. With the continuous development of terminal technology, multi-device collaboration technology can support a variety of different collaboration modes, such as: input device sharing, multi-screen collaboration (such as screen projection, application continuation, etc.) and other collaboration modes. The above-mentioned collaborative mode may also be referred to as a collaborative function or other names, which are not specifically limited here.

[0066] In the embodiment of the present application, when the terminal device 100 and at least one terminal device (such as the terminal device 200) perform cross-system and cross-device collaboration to achieve resource sharing and collaborative operation, the terminal device 100 and at least one terminal device (such as the terminal device 200) can be regarded as a "hyper terminal", and a hyper terminal page can be set on the terminal device 100 and / or the terminal device 200; taking the terminal device 200 as an example, the hyper terminal page on the terminal device 200 is used to achieve communication connection, task flow, collaborative mode, etc. between the terminal device 200 and other terminal devices (such as the terminal device 100). The hyper terminal page will be introduced in detail in the subsequent embodiments, which will not be repeated here.

[0067] Input device sharing: see Figure 1 , the terminal device 200 may have one or more input devices, such as a mouse and a keyboard; based on the above communication connection, the terminal device 200 may share the above one or more input devices with the terminal device 100, that is, the user may control the terminal device 100 through the above one or more input devices of the terminal device 200. For example, by moving the mouse of the terminal device 200, the user may control the cursor corresponding to the mouse to cross the boundary of the screen of the device and move to the screen of the terminal device 100, and then may manipulate the display content of the terminal device 100 through the mouse; when the mouse selects the text input box displayed by the terminal device 100, the user may enter text in the text input box through the keyboard of the terminal device 200. The embodiment of the present application does not specifically limit the implementation principle of the sharing of input devices. The embodiment of the present application does not specifically limit the display form of the cursor corresponding to the input device (such as a mouse and a touchpad) on the screen, and the cursor may also be invisible.

[0068] In the embodiments of the present application, an input device refers to a device that inputs data and information to an electronic device. The input device is one of the main devices for exchanging information between a user and an electronic device, including but not limited to a mouse, a keyboard, a camera, a handwriting tablet, a stylus, a remote control rod, a touch panel, a voice input device, and the like. The input device can generate input data based on detected user operations and input it into the electronic device. The generated input data can indicate the above-mentioned user operations (such as click operations, move operations), or can be information such as text, graphics, images, and sounds. The embodiments of the present application do not impose any restrictions on the type of input device and the input data generated by the input device. Reference Figure 1 , the input device can be an external device of the terminal device 200, which is connected to the terminal device 200 by wire or wirelessly. For example, a tablet computer can be wirelessly connected to a mouse; a laptop computer can be wired to a mouse. The input device can also be a built-in module of the terminal device 200, that is, a hardware module of the terminal device 200 itself. For example, the hardware modules of a laptop computer include a keyboard and a touchpad. The touchpad is used to control the cursor according to the input operation (such as movement and clicking) sensed by the user's fingers, and the touchpad can be regarded as a substitute for the mouse. The embodiment of the present application does not specifically limit the type of input device. It should be noted that in some usage scenarios, the terminal device 100 does not have an input device; in some usage scenarios, the terminal device 100 may also have an input device, and the terminal device 100 may also share the input device with the terminal device 200.

[0069] Screen projection: Based on the above communication connection, the terminal device 100 sends the preset screen display content to the terminal device 200 for display. Screen projection can include mirror projection and non-mirror projection. Among them, mirror projection refers to the terminal device 100 sending the real-time displayed screen content to the terminal device 200 for display; after projection, the display content of the projection window of the terminal device 200 is consistent with the screen content displayed by the terminal device 100. Non-mirror projection, the terminal device 100 sends the current screen display content or the display content of a specific application to the terminal device 200; after projection, the display content of the projection window of the terminal device 200 and the screen content displayed by the terminal device 100 may be different. Optionally, after mirror projection and non-mirror projection, the user can manipulate the display content from the terminal device 100 in the projection window of the terminal device 200, that is, the terminal device 100 can be manipulated through the projection window. The implementation principle of screen projection is not specifically limited in the embodiment of the present application. It should be noted that in some usage scenarios, the terminal device 200 can also project the screen to the terminal device 100.

[0070] In some screen projection scenarios, the terminal device 100 projects the user interface of the currently displayed application 1 to the terminal device 200. The projection content only includes the user interface of the application, and the user can only control the application through the projection window displayed by the terminal device 100.

[0071] In some screen projection scenarios, the terminal device 100 can project the currently displayed system interface (e.g., the main interface) or the user interface of the application (e.g., the chat page of the instant messaging application) to the terminal device 200, and the terminal device 200 displays the projected content in the projection window; then, the user can open other user interfaces of the terminal device 100 through the projection window, such as other system interfaces (e.g., the negative one screen) and the user interfaces of other applications (e.g., the user interface of text messages). It can be understood that the projected content in this scenario can include both the system interface and the application interface, and the user can freely control the terminal device 100 through the projection window.

[0072] Application continuation: Based on the above communication connection, the terminal device 100 can transfer the currently running application 2 to the terminal device 200 for continuous display. Specifically, the terminal device 100 can send the real-time usage data of application 2 to the terminal device 200; the terminal device 200 can locate the user's latest browsing position of application 2 on the terminal device 100 based on the above real-time usage data, so that the terminal device 200 can continue to display application 2. Application 2 of the terminal device 100 and the terminal device 200 and application 2 can be different versions of application 2. For example, the mobile phone sends the real-time usage data of the mobile version of the video application to the notebook, and the real-time usage data indicates the video page 1 that the user is playing in the video application and the current playback time of video 1 in video page 1, such as playback time 1 of video 1; the laptop runs the computer version of video application 1, and automatically starts playing video 1 on video page 1 from playback time 1 of video 1 according to the above real-time usage data. The implementation principle of application continuation is not specifically limited in the embodiment of the present application. It should be noted that in some usage scenarios, the terminal device 100 can also perform application continuation on the application running on the terminal device 200.

[0073] In the embodiment of the present application, different collaborative modes correspond to different start-up conditions, for example, input device sharing corresponds to start-up condition 1, and multi-screen collaboration corresponds to start-up condition 2. When the user initially turns on the collaborative mode, the terminal device 100 / terminal device 200 detects the start-up condition currently satisfied by the device status of the terminal device 100, and then triggers the terminal device 200 and the terminal device 100 to start the collaborative mode corresponding to the start-up condition. In the embodiment of the present application, the user can trigger the start of any collaborative mode through the terminal device 100, and can also trigger the start of any collaborative mode through the terminal device 200, which is not specifically limited here. In this way, the user does not need to manually select the collaborative mode, and the collaborative mode can be adaptively selected according to the device status of the terminal device 100 to match the real-time needs of the user.

[0074] In some embodiments, based on the above-mentioned communication connection, when the terminal device 100 and the terminal device 200 are in the first collaborative mode, if the terminal device 100 / terminal device 200 detects that the device state of the terminal device 100 meets the start condition of the second collaborative mode, the first collaborative mode can be automatically switched to the second collaborative mode, that is, the first collaborative mode is stopped and the second collaborative mode is started. In some embodiments, based on the above-mentioned communication connection, when the terminal device 100 and the terminal device 200 are in the first collaborative mode, the user can also manually switch the first collaborative mode to the second collaborative mode. The embodiment of the present application does not specifically limit the operation of manual switching by the user. In the above embodiment, the first collaborative mode and the second collaborative mode cannot coexist.

[0075] In some embodiments, based on the above communication connection, when the terminal device 100 and the terminal device 200 perform the first collaborative mode, if the terminal device 100 / terminal device 200 detects that the terminal device 100 meets the start condition of the second collaborative mode, the second collaborative mode can be started, that is, the first collaborative mode and the second collaborative mode are performed at the same time. In some embodiments, based on the above communication connection, when the terminal device 100 and the terminal device 200 perform the first collaborative mode, the user can also manually start the second collaborative mode. The embodiment of the present application does not specifically limit the operation manually started by the user. In the above embodiment, the first collaborative mode and the second collaborative mode can coexist.

[0076] For example, the first collaborative mode is input device sharing, and the second collaborative mode is multi-screen collaboration; or, the first collaborative mode is multi-screen collaboration, and the second collaborative mode is input device sharing.

[0077] It should be noted that the embodiment of the present application does not limit the type of the above-mentioned communication connection, and the terminal device 100 and the terminal device 200 can transmit data or instructions through the established communication connection.

[0078] In some embodiments, the terminal device 100 and the terminal device 200 may be directly connected via a short-range wireless communication connection or a local wired connection.

[0079] Exemplarily, the short-range communication module involved in the above-mentioned short-range wireless communication connection may include a wireless local area network (WLAN) communication module, an ultra wide band (UWB) communication module, an infrared communication module, a Bluetooth communication module, a near field communication (NFC) communication module, a ZigBee communication module, etc. Among them, the Bluetooth module can provide solutions including one or more Bluetooth communications in classic Bluetooth (Bluetooth 2.1) or Bluetooth low energy (BLE), and the WLAN module can provide solutions including one or more WLAN communications in wireless fidelity peer-to-peer (Wi-Fi P2P), wireless fidelity local area networks (Wi-Fi LAN) or wireless fidelity software access point (Wi-Fi softAP). In some embodiments, Wi-Fi P2P refers to allowing devices in a wireless network to connect to each other in a point-to-point manner without going through a wireless router. The system is also called wireless fidelity direct (Wi-Fi direct). In one implementation, the terminal device 100 can detect and scan electronic devices near the terminal device 100 by transmitting signals through a short-range communication module (such as a Bluetooth communication module), so that the terminal device 100 can find nearby electronic devices (such as terminal device 200) through a short-range wireless communication protocol, establish a short-range wireless communication connection (such as a Bluetooth connection) with the nearby electronic devices, and transmit data to the nearby electronic devices.

[0080] Exemplarily, the local wired connection may include a universal serial bus (USB) connection, a high definition multimedia interface (HDMI) connection, a display port (DP) connection, and the like.

[0081] In some embodiments, the terminal device 100 and the terminal device 200 can perform indirect communication over a long distance through at least one electronic device in a communication network, and the communication network includes a local area network (LAN) and / or a wide area network (WAN). In one implementation, the terminal device 100 and the terminal device 200 can be connected to a local area network (LAN) through at least one electronic device 300 based on a wired or wireless fidelity (WiFi) connection. For example, the electronic device 300 may include a third-party device such as a router, a gateway, and an intelligent device controller. In one implementation, the terminal device 100 and the terminal device 200 can also be indirectly connected through at least one electronic device 400 in a wide area network (such as the Internet). For example, the electronic device 400 may include one or more hardware servers, cloud servers implanted in a virtualized environment, and other devices. It can be understood that through the electronic device 300 and / or the electronic device 400, the terminal device 100 and the terminal device 200 indirectly perform wireless communication connection and data transmission.

[0082] In the embodiment of the present application, the terminal device 100 and the terminal device 200 can log in to the same account, and then achieve communication connection through the Internet based on the same account. Alternatively, the terminal device 100 and the terminal device 200 can also log in to different accounts, but connect by binding. For example, a mobile phone and a laptop can log in to different accounts, and the mobile phone binds the laptop to itself in the device management application, and then controls the communication connection through the device management application.

[0083] In some embodiments, when two terminal devices are mutually trusted devices (for example, the two terminal devices have been matched or connected before), the two terminal devices can automatically establish a communication connection and then exchange data without the user having to manually connect or match again, saving time and effort. For example, electronic devices that have been paired with Bluetooth, electronic devices that have been connected to a shared hotspot, and electronic devices that have established a Wi-Fi P2P connection can be confirmed as trusted devices.

[0084] In some embodiments, input device sharing and multi-screen collaboration use the same communication resources (eg, WiFi P2P communication), and the terminal device 100 and the terminal device 200 can only implement one of input device sharing and multi-screen collaboration at the same time.

[0085] In some embodiments, input device sharing and multi-screen collaboration can use different communication resources, and the terminal device 100 and the terminal device 200 can simultaneously implement input device sharing and multi-screen collaboration. For example, input device sharing uses WiFi P2P communication, and multi-screen collaboration uses Bluetooth communication; or, input device sharing and multi-screen collaboration use Bluetooth communication in time division; or, input device sharing and multi-screen collaboration use Bluetooth communication in frequency division.

[0086] Each terminal device in the communication system 10 can be equipped with system, system, system, HarmonyOS or other types of operating systems. The operating systems of various terminal devices can be the same or different, and this application does not limit this.

[0087] It should be understood that Figure 1 The system structure diagram of the communication system provided in the embodiment of the present application is merely a schematic diagram, and does not constitute a specific limitation on the communication system 10. In an actual service scenario, the communication system 10 may include more or fewer devices than those shown in the figure.

[0088] The following describes a structure of a terminal device 100 provided in an embodiment of the present application. The structure of the terminal device 200 can refer to the relevant description of the terminal device 100, which will not be described in detail later.

[0089] Figure 2AThe structure diagram of the terminal device 100 is shown. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0090] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0091] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0092] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0093] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0094] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0095] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the terminal device 100.

[0096] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0097] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0098] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0099] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 100.

[0100] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0101] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0102] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0103] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0104] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0105] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0106] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0107] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0108] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0109] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0110] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0111] The terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0112] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0113] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0114] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0115] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0116] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0117] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0118] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0119] The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0120] Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation DDR SDRAM is generally referred to as DDR5 SDRAM), etc.; non-volatile memory may include disk storage devices and flash memory.

[0121] Flash memory can be divided into NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle; single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. according to the storage unit potential level; universal flash storage (UFS), embedded multi media card (eMMC), etc. according to the storage specification.

[0122] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0123] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0124] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the terminal device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external non-volatile memory.

[0125] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0126] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0127] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0128] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0129] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

[0130] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile electronic device platform standard interface or a Cellular Telecommunications Industry Association standard interface.

[0131] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0132] The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., X, Y, and Z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0133] It should be noted that the reference coordinate system of the gyroscope sensor is usually the geographic coordinate system. Figure 2B The three-axis (i.e., X-axis, Y-axis, and Z-axis) coordinate system shown is a geographic coordinate system exemplarily shown in the embodiment of the present application. Figure 2BAs shown, the origin O of the geographic coordinate system can be located at the point where the carrier (i.e., the device containing the gyroscope sensor, such as the terminal device 100) is located, the X-axis points to the east (E) along the local latitude, the Y-axis points to the north (N) along the local meridian, and the Z-axis points upward along the local geographic vertical line, and forms a right-handed rectangular coordinate system with the X-axis and the Y-axis. Among them, the plane formed by the X-axis and the Y-axis is the local horizontal plane, and the plane formed by the Y-axis and the Z-axis is the local meridian plane.

[0134] For example, taking a smartphone as an example, Figure 2C The three-axis (i.e., Xs axis, Ys axis and Zs axis) coordinate system shown is a device coordinate system of the terminal device 100 shown in the embodiment of the present application, wherein the origin of the device coordinate system can be a preset point (e.g., the center of mass) of the terminal device 100, the Xs axis points from the center of mass of the terminal device 100 to the right side of the terminal device, the Ys axis points from the center of mass of the terminal device 100 to the top of the terminal device 100, the Ys axis is perpendicular to the Xs axis, and the plane formed by the Xs axis and the Ys axis is parallel to the screen of the terminal device 100; and the Zs axis points from the center of mass of the terminal device to the screen of the terminal device, and is perpendicular to the plane formed by the Xs axis and the Ys axis.

[0135] The posture of the terminal device 100 can be determined by three posture angles: pitch, yaw, and roll. In one implementation, the pitch angle can be the angle between the Ys axis of the device coordinate system and the local horizontal plane; the azimuth angle can be the angle between the projection of the Ys axis of the device coordinate system on the local horizontal plane and the Y axis of the ground coordinate system; the roll angle can be the angle between the Zs axis of the device coordinate system and the vertical plane of the Ys axis, and the vertical plane of the Ys axis can be understood as the plane formed by the Ys axis and the Z axis of the geographic coordinate system. In the embodiment of the present application, the terminal device 100 can determine the three posture angles of the terminal device 100 based on the data collected by the gyroscope, and then determine the current posture of the terminal device 100. In the embodiment of the present application, the method for determining the posture of the terminal device 100 is not specifically limited.

[0136] In the embodiment of the present application, the collaborative mode currently applicable to the terminal device 100 and the terminal device 200 can be determined according to the posture of the terminal device 100. Taking a mobile phone and a laptop computer as an example, when the screen of the mobile phone is not inclined to be upright or facing the user, it is not convenient for the user to view the screens of the mobile phone and the laptop computer at the same time, and it is not convenient to control the screen of the mobile phone through the mouse of the laptop computer, that is, it is not convenient to share the input device. At this time, multi-screen collaboration can be used. Correspondingly, when the mobile phone is inclined to be upright and the screen of the mobile phone faces the user, it is convenient for the user to view the screens of the mobile phone and the computer at the same time, and it is convenient to control the screen of the mobile phone through the mouse of the laptop computer, that is, it is convenient to share the input device.

[0137] In the embodiment of the present application, it can be determined whether the posture of the terminal device 100 is inclined to be upright based on the data collected by the gyroscope sensor 180B. In one implementation, the terminal device 100 is inclined to be upright when: the pitch angle of the terminal device 100 is greater than the angle threshold 1 (for example, 60°), less than or equal to the angle threshold 2 (for example, 100°), and the absolute value of the angle between the Xs axis of the terminal device 100 and the horizontal plane is less than or equal to the angle threshold 3 (for example, 15°); or, the roll angle of the terminal device 100 is greater than the angle threshold 1 (for example, 60°), less than or equal to the angle threshold 2 (for example, 100°), and the absolute value of the angle between the Ys axis of the terminal device 100 and the horizontal plane is less than or equal to the angle threshold 3 (for example, 15°). In another implementation, the terminal device 100 is inclined to be upright means that: the angle between the screen of the terminal device 100 (i.e., the plane formed by the Xs axis and the Ys axis of the terminal device 100) and the horizontal plane is greater than the angle threshold 4 (e.g., 60°) and less than or equal to the angle threshold 5 (e.g., 100°), and the absolute value of the angle between the Xs axis or the Ys axis of the terminal device 100 and the horizontal plane is less than or equal to the angle threshold 6 (e.g., 15°). In the embodiment of the present application, the implementation method of detecting whether the posture of the mobile phone is inclined to be upright is not specifically limited.

[0138] For ease of description, the present application refers to the screen of the terminal device 100 being in a posture that is convenient for the user to manipulate through the input device as a convenient posture. For example, the convenient posture means that the terminal device 100 is inclined to stand upright, and the screen of the terminal device 100 faces the user.

[0139] In some embodiments, when the terminal device 200 shares the input device with the terminal device 100, if the user changes the posture of the terminal device 100 so that the screen of the terminal device 100 is out of the convenient posture, the terminal device 100 and the terminal device 200 can automatically switch the input device sharing to multi-screen collaboration (such as screen projection, application continuation) without user operation. In this way, when it is inconvenient for the user to control the terminal device 100 through the input device of the terminal device 200, the user can continue to view and control the display content from the terminal device 100 through the screen projection window or application continuation window in time.

[0140] Conversely, in some embodiments, when the terminal device 100 and the terminal device 200 perform multi-screen collaboration, if the user changes the posture of the terminal device 100 so that the screen of the terminal device 100 switches to a convenient posture, the terminal device 100 and the terminal device 200 can automatically switch the multi-screen collaboration to input device sharing without user operation. In this way, when the screen of the terminal device 100 switches to a posture that is convenient for control through the input device, the user can also view and control the terminal device 100 through the input device of the terminal device 200 in a timely manner.

[0141] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0142] The magnetic sensor 180D includes a Hall sensor, and the terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case.

[0143] The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal device 100 in various directions (generally three axes). When the terminal device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0144] The distance sensor 180F is used to measure the distance. The terminal device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0145] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light outward through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can use the proximity light sensor 180G to detect that the user holds the terminal device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, and pocket mode automatically unlocks and locks the screen.

[0146] In an embodiment of the present application, when an anti-mistaken touch scenario is detected (for example, a scenario in which the terminal device 100 is placed in a pocket or bag), the terminal device 100 can turn on the anti-mistaken touch function to prevent the user from accidentally touching the screen of the terminal device 100. If the user accidentally touches the screen, the device may be mislocked or the application may be accidentally clicked, thereby causing misoperation, such as making a wrong call, deleting a text message, or logging into the network and using traffic, which may cause the loss of important information in severe cases. Turning on the anti-mistaken touch function in the anti-mistaken touch scenario can provide users with a friendly operating environment and reduce device power consumption. In an embodiment of the present application, the above-mentioned anti-mistaken touch scenario and the method of detecting the anti-mistaken touch scenario are not specifically limited; in addition, the user can also manually turn on the anti-mistaken touch function of the terminal device 100.

[0147] In some embodiments, the proximity light sensor 180G can be used to determine whether the anti-false touch function can be turned on. For example, when the proximity light sensor 180G detects that the screen of the terminal device 100 is blocked within a preset distance, the terminal device 100 turns on the anti-false touch function. The embodiment of the present application can also be combined with other sensors (such as ultrasonic sensors) to detect whether the screen of the terminal device 100 is blocked within a preset distance, which is not specifically limited here.

[0148] The ambient light sensor 180L is used to sense the ambient light brightness. The terminal device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in a pocket or bag to prevent accidental touch. In one embodiment, the ambient light sensor 180L and the proximity light sensor 180G can be coupled to determine whether the terminal device 100 has turned on the anti-false touch function. For example, when the ambient light sensor 180L detects that the ambient light brightness is lower than the first brightness value, such as 10 lux (lx), and the proximity light sensor 180G detects that the screen of the current terminal device 100 is in an obstructed state, the terminal device 100 will turn on the anti-false touch function.

[0149] In some embodiments, when the terminal device 200 shares the input device with the terminal device 100, if it is detected that the terminal device 100 is in an anti-mistouch scenario or the anti-mistouch function is turned on, no user operation is required, and the terminal device 100 and the terminal device 200 can automatically switch the input device sharing to multi-screen collaboration.

[0150] In some embodiments, when the terminal device 100 is projecting the screen to the terminal device 200, if it is detected that the terminal device 100 is out of the anti-mistouch scenario or the anti-mistouch function is turned off, no user operation is required, and the terminal device 100 and the terminal device 200 can automatically switch the multi-screen collaboration to input device sharing.

[0151] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0152] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy.

[0153] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal device 100, which is different from the position of the display screen 194.

[0154] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of a human vocal part.

[0155] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.

[0156] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0157] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.

[0158] The SIM card interface 195 is used to connect a SIM card.

[0159] For example, taking the terminal device 100 equipped with HarmonyOS as an example, Figure 3 A software system architecture of the terminal device 100 is shown.

[0160] like Figure 3 As shown, as an example, HarmonyOS200 adopts a layered architecture. The layered architecture divides HarmonyOS into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, HarmonyOS is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the system service layer, and the kernel layer. In other embodiments, HarmonyOS200 can also be divided into other numbers of hierarchical structures, which are not limited here.

[0161] Among them, the application layer can include a series of applications (Application, APP) such as system applications and third-party applications (or called extended applications). System applications may include applications installed by default on terminal devices such as desktop, control bar, settings, and phone. Extended applications can be non-essential applications developed and designed by the manufacturer of the terminal device, such as device managers, music, notes, instant messaging applications, and other applications. Third-party non-system applications can be developed by other manufacturers, but can run applications in the Hongmeng system, such as games, navigation, social or shopping applications.

[0162] In some embodiments, the applications in the application layer of the terminal device 100 may include two categories of applications. The first category is applications that can be connected to the terminal device 200. The applications are applications that can be installed by both the terminal device 100 and the terminal device 200. They can be system applications or third-party applications, such as video, navigation, and instant messaging. The second category is applications that cannot be connected to the terminal device 200, such as text messages, phone calls, desktops, etc. in system applications, or third-party applications that cannot be installed by the terminal device 200.

[0163] In some embodiments of the present application, applications may be atomized. Specifically, applications may be composed of one or more capabilities (also known as atomic capabilities, etc.). Capabilities include, but are not limited to, feature capabilities (FA, also known as metaprograms) and meta capabilities (AA, also known as metaservices). FA has a user interface (UI) and provides the ability to interact with users; AA has no UI interface and provides the ability to run tasks in the background and a unified data access abstraction. FA and AA may also have other names, which are not limited in the embodiments of the present application.

[0164] The application framework layer can provide application programming interface (API) and programming framework for the applications in the application layer, for example, providing user interface (UI) framework, capability framework, and user program framework.

[0165] The system service layer is the core capability set of the system, providing services to the applications in the application layer through the framework layer. This layer includes the following parts:

[0166] Basic capability subsystem set: provides basic capabilities for the operation, scheduling, migration and other operations of distributed applications on multiple devices, which may include distributed soft bus (or, can also be described as soft bus subsystem), distributed data management module, distributed task scheduling module, public basic library (not shown in the figure), multi-mode input, graphics, security, artificial intelligence (AI) and other subsystems.

[0167] Among them, the distributed soft bus is a logical channel that can be used for communication between devices. In the embodiment of the present application, multiple electronic devices (such as terminal device 100 and terminal device 200) can achieve communication connection, data transmission, hardware mutual assistance, and resource sharing through a distributed soft bus, distributed device virtualization, distributed data management, and distributed task scheduling, such as collaboratively realizing input device sharing, screen projection, and application continuation.

[0168] The multi-mode input subsystem provides input services for the terminal device 100, monitors the user's input operations on the terminal device 100, such as receiving input events through input devices such as a touch screen, mouse, and keyboard, and processes these original input events before dispatching them. In some embodiments, when the input device of the terminal device 200 is shared with the terminal device 100, the multi-mode input subsystem of the terminal device 200 can be used to monitor the input event (such as input event 1) of the input device of the terminal device 200 (such as a mouse) for the terminal device 100, and can share the input event 1 with the terminal device 100; the multi-mode input subsystem of the terminal device 100 can process and distribute the monitored input event 1; in this way, the user can control the terminal device 100 through the input device of the terminal device 200.

[0169] In some embodiments, after the input device of the terminal device 200 is shared with the terminal device 100, when the multi-mode input subsystem of the terminal device 200 monitors the input device (such as a mouse) of the terminal device 200 for the input event 1 of the terminal device 100, the multi-mode input subsystem of the terminal device 100 is shared with the multi-mode input subsystem of the terminal device 100 through the distributed soft bus, and the multi-mode input subsystem of the terminal device 100 processes and distributes the input event 1.

[0170] In some embodiments, the distributed data management module can be used to receive and record application state data, and synchronize the application state data to the distributed system (e.g. Figure 1 In other terminal devices of the communication system shown in the figure).

[0171] In some embodiments, after the input device of the terminal device 200 is shared with the terminal device 100, when the multi-mode input subsystem of the terminal device 200 monitors the input event 1 of the input device (such as a mouse) of the terminal device 200 for the terminal device 100, the input event 1 can be uploaded to the distributed data management module, and then the input event 1 can be shared with the distributed data management module of the terminal device 100; the multi-mode input subsystem of the terminal device 100 monitors the input event 1 from the distributed data management module, and processes and distributes the input event 1.

[0172] The drag framework can be used to realize the dragging of pictures, files, user interfaces and other contents from the screen of the terminal device 200 to the screen of the present device through an input device such as a mouse, or dragging pictures, files, user interfaces and other contents from the screen of the present device to the screen of the terminal device 200 in the input device sharing mode. For example, when the user interface of the present device is dragged to the screen of the terminal device 200, the drag framework of the present device can obtain an interface snapshot of the interface, and directly send it to the drag framework of the terminal device 200 through the distributed soft bus, or share it with the drag framework of the terminal device 200 through the distributed data management module; the drag framework of the terminal device 200 can trigger the display mouse to drag the above interface snapshot to move on the screen of the terminal device 200.

[0173] The crossing service is used to sense mouse events such as pressing the mouse, moving the mouse, and releasing the mouse, and to process the above mouse events to enable the cursor corresponding to the mouse to cross between the screen of the terminal device 200 and this device.

[0174] Basic software service subsystem set: provides public and general software services for the system, consisting of event notification, telephone, multimedia, DFX, MSDP&DV and other subsystems.

[0175] Enhanced software service subsystem set: provides the system with differentiated capability-enhanced software services for different devices, and consists of subsystems such as smart screen proprietary services, wearable proprietary services, and IoT proprietary services.

[0176] Hardware service subsystem set: provides hardware services, consisting of subsystems such as location services, biometric recognition, wearable proprietary hardware services, and IoT proprietary hardware services.

[0177] According to the deployment environment of different device forms, the basic software service subsystem set, enhanced software service subsystem set, and hardware service subsystem set can be tailored according to the subsystem granularity, and each subsystem can be tailored according to the functional granularity.

[0178] The kernel layer is the layer between hardware and software. The kernel layer is divided into kernel subsystem and driver subsystem.

[0179] Kernel subsystem: adopts multi-core design. The kernel abstract layer (KAL) on the kernel subsystem shields the differences between multiple cores and provides basic kernel capabilities to the upper layer, including process / thread management, memory management, file system, network management, and peripheral management.

[0180] The driver subsystem is the system's driver framework (HDF), and is also the foundation for the openness of the distributed system hardware ecosystem, providing unified peripheral access capabilities and driver development and management frameworks. The kernel layer includes at least display drivers, camera drivers, audio drivers, and sensor drivers.

[0181] Figure 3 The system architecture shown does not constitute a limitation on the system architecture of the terminal device. The system architecture of the terminal device 100 may have a larger system architecture than the above-mentioned system architecture. Figure 3 The structure shown may be more or less system architecture structures, or other forms of system architecture, which are not specifically limited here.

[0182] In combination with the aforementioned communication system, hardware structure and software system architecture, the embodiment of the present application provides a multi-device collaboration method, in which the terminal device 100 and the terminal device 200 can select and implement an appropriate collaboration mode in a plurality of collaboration modes according to the real-time status of the terminal device 100 and / or user operation, so as to meet the real-time needs of the user as much as possible. The above-mentioned plurality of collaboration modes may include: input device sharing, screen projection and application continuation.

[0183] For ease of understanding, the following specifically introduces the multi-device collaboration method provided by the embodiments of the present application in combination with interface embodiments 1 to 4. For ease of description, the subsequent embodiments are described by taking the terminal device 100 as a mobile phone 100 and the terminal device 200 as a computer 200 as an example.

[0184] Interface Example 1

[0185] In the first embodiment of the interface, different device states of the mobile phone 100 may correspond to different collaborative modes. When the collaborative mode is initially started, under different device states of the mobile phone 100, the user can trigger the mobile phone 100 and the computer 200 to enter different collaborative modes through a preset operation 1.

[0186] In some embodiments, after detecting a preset operation 1 acting on the mobile phone 100 / computer 200, the mobile phone 100 / computer 200 detects the device status of the mobile phone 100; when the device status of the mobile phone 100 meets the start condition 1, the mobile phone 100 / computer 200 can trigger the start of input device sharing; when the device status of the mobile phone 100 meets the start condition 2, the mobile phone 100 / computer 200 can trigger the start of multi-screen collaboration (such as screen casting, application continuation). It can be understood that in the embodiments of the present application, the initial startup collaboration mode can be triggered by the mobile phone 100, and the initial startup collaboration mode can also be triggered by the computer 200, which is not specifically limited here.

[0187] For example, FIG. 4A to FIG. 4I It shows that the user triggers two terminal devices to start the first collaborative mode through the computer 200.

[0188] Figure 4A and Figure 4B The user interface of the computer 200 starting the hyperterminal page is shown.

[0189] For example, Figure 4A 1 shows a main interface 11 of a computer 200, and the status bar of the main interface 11 includes a hyperterminal icon 101. Figure 4A and Figure 4B As shown, the computer 200 detects an input operation on the hyperterminal icon 101 (eg, a click operation performed by the mouse when the cursor corresponding to the mouse moves to the hyperterminal icon 101 ), and the computer 200 displays the hyperterminal page 12 .

[0190] In the embodiment of the present application, the user can also open the hyperterminal page 12 in other ways. For example, the user can open the hyperterminal page 12 through the control center, notification bar, desktop card or an application (such as a smart home application, a setting application) of the computer 200. The embodiment does not limit the way to open the hyperterminal page 12. Figure 4B The hyperterminal page 12 shown is an example provided by an embodiment of the present application. The hyperterminal page 12 may include more or fewer interface elements, which are not specifically limited here.

[0191] The hyperterminal page 12 may include a hyperterminal icon 102, a more icon 103, a zoom-out icon 104, an enlargement icon 105, and a close icon 106. The more icon 103 is used to trigger more functions of the hyperterminal, the zoom-out icon 104 is used to minimize the hyperterminal page 12, the enlargement icon 105 is used to maximize the hyperterminal page 12, and the close icon 106 is used to close the hyperterminal page 12.

[0192] The hyperterminal page 12 also includes an icon 107 of the computer 200 and an icon of at least one candidate device detected by the computer 200 for collaborative connection, such as an icon 108 of the mobile phone 100 .

[0193] For example, the icon 107 of the computer 200 is located in the preset area 1 (eg Figure 4B The icon of the at least one candidate device may be located in a preset area 2 surrounding the icon 107 of the computer 200 (eg, Figure 4B The embodiment of the present application does not specifically limit the shape and position of the above-mentioned preset area 1 and the preset area 2. The embodiment of the present application does not specifically limit the distribution mode of the above-mentioned at least one candidate device in the preset area 2.

[0194] In some embodiments, the distance of the icon 108 of the mobile phone 100 relative to the icon 107 of the computer 200 is determined according to the distance of the mobile phone 100 relative to the computer 200; the orientation of the icon 108 of the mobile phone 100 relative to the icon 107 of the computer 200 is determined according to the orientation of the mobile phone 100 relative to the computer 200. For example, when the mobile phone 100 is located on the left side of the computer 200, the icon 108 of the mobile phone 100 is also located on the left side of the icon 107 of the computer 200. In the embodiments of the present application, the method for measuring the relative distance and relative orientation between the computer 200 and the mobile phone 100 is not specifically limited.

[0195] In some embodiments, upon receiving and responding to a preset operation 1 performed by the user on the icon 108 of the mobile phone 100 (for example, dragging the icon 108 close to the icon 107 of the computer 200), the computer 200 requests a collaborative connection with the mobile phone 100, and detects the device status of the mobile phone 100 after the collaborative connection; if it is detected that the device status of the mobile phone 100 satisfies the startup condition, the computer 200 automatically instructs the mobile phone 100 to perform a collaborative mode corresponding to the startup condition.

[0196] In some embodiments, the hyperterminal page 12 may further include prompt information 109, which is used to prompt that a collaborative connection can be established by dragging the device icon to the local icon. Figure 4B and Figure 4C, after detecting that the user selects the icon 108 of the mobile phone 100 (for example, by long pressing the left mouse button to select the icon 108), dragging the icon 108 close to the icon 107 of the computer 200, and stopping selecting the icon 108 (for example, by stopping long pressing the left mouse button to stop selecting the icon 108), the computer 200 adsorbs the icon 108 of the mobile phone 100 to the edge of the icon 107 of the computer 200, and establishes a collaborative connection with the mobile phone 100; after successfully establishing a collaborative connection, a prompt message may be displayed to remind the user that the mobile phone 100 and the computer 200 have established a collaborative connection. In one implementation, when the computer 200 detects that the distance between the icon 108 of the mobile phone 100 and the icon 107 of the computer 200 is less than or equal to the distance 1, the computer 200 and the mobile phone 100 are collaboratively connected.

[0197] In some embodiments, based on the preset operation 1, after the computer 200 and the mobile phone 100 establish a collaborative connection, the computer 200 may send a request message 1 to the mobile phone 100, and the request message 1 is used to instruct the mobile phone 100 to detect whether the device state of the mobile phone 100 meets the start-up conditions corresponding to each collaborative mode. Taking the satisfaction of start-up condition 1 as an example, when the mobile phone 100 detects that the device state of the mobile phone 100 meets the start-up condition 1 according to the relevant parameters of the device state, the computer 200 sends a response message 1 to the computer 200, and the response message 1 indicates the start-up condition 1 or the collaborative mode corresponding to the start-up condition 1; according to the response message 1, the computer 200 determines to instruct the mobile phone 100 to perform the collaborative mode corresponding to the start-up condition 1.

[0198] In some embodiments, based on preset operation 1, after the computer 200 and the mobile phone 100 establish a collaborative connection, the computer 200 may send a request message 2 to the mobile phone 100, and the request message 2 is used to obtain relevant parameters of the device status of the mobile phone 100; after the computer 200 receives the above-mentioned relevant parameters fed back by the mobile phone 100, it detects whether the device status of the mobile phone 100 meets the startup conditions corresponding to each collaborative mode according to the above-mentioned relevant parameters; when it is determined that the device status of the mobile phone 100 meets the startup conditions, the computer 200 determines to instruct the mobile phone 100 to perform the collaborative mode corresponding to the startup condition.

[0199] In some embodiments, based on preset operation 1, after the computer 200 and the mobile phone 100 establish a collaborative connection, the computer 200 may send a request message 3 to the mobile phone 100, and the request message 3 is used to instruct the mobile phone 100 to perform multi-device collaboration; when the mobile phone 100 detects that the device status of the mobile phone 100 meets the startup condition according to the request message 3, the mobile phone 100 determines to instruct the computer 200 to perform the collaborative mode corresponding to the startup condition.

[0200] In the embodiment of the present application, the developer can define the starting conditions of each collaborative mode according to the actual application scenario (such as the device type of the device performing multi-device collaboration), which is not specifically limited here.

[0201] In some embodiments, the computer 200 receives and responds to a preset operation 1. If it detects that the device status of the mobile phone 100 satisfies the start condition 1, the computer 200 shares the input device (such as a mouse) with the mobile phone 100, and the user can control the mobile phone 100 through the input device of the computer 200.

[0202] For example, Figure 4C As shown, the device status of mobile phone 100 meets the start condition 1. After computer 200 shares the mouse with mobile phone 100, if the cursor follows the mouse to the border of the computer 200 screen and then continues to move to the outside of the screen, the cursor can cross from the screen border of computer 200 to the screen border of mobile phone 100 and continue to move on the screen of mobile phone 100; at this time, the screen of computer 200 does not display the cursor, but the screen of mobile phone 100 displays the cursor. When the cursor moves to the screen of mobile phone 100, the user can control mobile phone 100 to perform various operations through the mouse. Figure 4C and Figure 4D As shown, the main interface 13 of the mobile phone 100 includes an icon 111 of an instant messaging application; after the cursor moves to the icon 111, the mobile phone 100 receives and responds to the user's operation of double-clicking the icon 111 with the mouse, and the instant messaging application corresponding to the icon 111 is run in the foreground, and the homepage 14 of the instant messaging application is displayed. Similarly, after the cursor crosses the screen of the mobile phone 100, if the cursor follows the mouse to the screen boundary of the mobile phone 100 and then continues to move to the outside of the screen, the cursor can cross from the screen boundary of the mobile phone 100 back to the screen of the computer 200.

[0203] In some embodiments, following the movement of the mouse, the cursor may pass through the boundary 1 of the screen of the computer 200 and pass through the boundary 2 of the screen of the mobile phone 100. Wherein, the boundary 1 and the boundary 2 may be any one of the left boundary, the right boundary, the upper boundary, and the lower boundary of the screen. In one implementation, "boundary 1 and boundary 2" are associated with the "relative position of the mobile phone 100 and the computer 200". For example, Figure 4C As shown, the mobile phone 100 is located on the left side of the computer 200, and the user moves the mouse to the left, so that the cursor passes through the left border of the screen of the computer 200 and passes through the right border of the screen of the mobile phone 100. The following embodiments take the left border of the screen of the computer 200 and the right border of the screen of the mobile phone 100 as examples for exemplary description.

[0204] In some embodiments, the above-mentioned startup condition 1 includes one or more of the following: "not in an anti-mistouch scenario (for example, placed in a bag or pocket), the anti-mistouch function is not turned on, the screen is facing the user (for example, facing the desktop, facing the sky, facing away from the user), the posture tends to be upright, and the distance between the mobile phone 100 and the computer 200 is less than distance 2".

[0205] In some embodiments, the above-mentioned starting condition 1 includes: the posture of the mobile phone 100 tends to be upright, and the distance between the mobile phone 100 and the computer 200 is less than the distance 2.

[0206] In one implementation, the mobile phone 100 stores the face of a preset user; the front camera of the mobile phone 100 captures image 1, and if it is detected that image 1 includes the face of the preset user, it is determined that the screen of the mobile phone 100 faces the user. In one implementation, the mobile phone 100 stores the face of a preset user; the front camera of the mobile phone 100 captures image 1, and if it is detected that image 1 includes the face of the preset user, and it is determined based on the facial features of the face that the angle between the face plane and the screen is less than an angle threshold 7 (e.g., 55°), it is determined that the screen of the mobile phone 100 faces the user. The above-mentioned face plane can be determined based on the facial features of the face.

[0207] It should be noted that the present application embodiment does not specifically limit the distance measurement method between the two devices, such as ultrasonic ranging algorithm, time of flight ranging algorithm, GPS positioning method, etc. The implementation method of detecting whether the posture of the mobile phone is inclined to be upright can refer to the above embodiment and will not be repeated here.

[0208] In some embodiments, the computer 200 shares the input device with the mobile phone 100, including: Figure 4C, the computer 200 sends a message 1 to the mobile phone 100, which indicates that the input device is shared with the mobile phone 100, and the kernel layer of the computer 200 creates a virtual display screen corresponding to the mobile phone 100, and the mouse cursor of the computer 200 can move from the physical display screen to the virtual display screen; the multi-mode input subsystem of the mobile phone 100 adds the input device information (such as device ID) corresponding to each input device shared by the computer 200 based on the message 1, and the multi-mode input subsystem can monitor the input events of the input device shared by the computer 200 according to the above input device information. When the cursor moves to the virtual display screen corresponding to the mobile phone 100, the multi-mode input subsystem of the computer 200 obtains the input event of the shared input device (such as mouse) monitored, and converts the above input event into the input event of the mobile phone 100 (such as mouse movement event, mouse click event) according to the event mapping table, and transmits it to the mobile phone 100; the multi-mode input subsystem of the mobile phone 100 can monitor the above input event transmitted by the computer 200, and process and distribute the above input event, and then the mobile phone 100 can respond to the above input event, such as displaying the cursor, moving the cursor, etc. In one implementation, the virtual display screen and the physical display screen of the mobile phone 100 have the same resolution, and each pixel corresponds to each other, so the coordinates of the cursor on the virtual display screen are the same as the coordinates of the cursor on the mobile phone 100, and the coordinates of the cursor on the virtual display screen can be directly sent to the mobile phone 100 through an input event without complicated coordinate conversion. The embodiment of the present application does not specifically limit the implementation of input device sharing between the computer 200 and the mobile phone 100.

[0209] In some embodiments, the computer 200 receives and responds to a preset operation 1. If it is detected that the device status of the mobile phone 100 satisfies the start condition 2, the computer 200 instructs the mobile phone 100 to project the preset screen display content to the computer 200.

[0210] For example, Figure 4E As shown, the device status of the mobile phone 100 meets the start condition 2, and the mobile phone 100 uses the screen projection technology to send the current main interface 13 to the computer 200, and the computer 200 displays the main interface 13 from the mobile phone 100 in the screen projection window 15. Optionally, the computer 200 can control the display content in the screen projection window. Figure 4E and Figure 4F As shown, the main interface 13 displayed in the projection window 15 includes an icon 111 of an instant messaging application; after the cursor moves to the icon 111, it receives and responds to the user's operation of double-clicking the icon 111 with the mouse, and the projection window 15 displays the homepage 14 of the instant messaging application corresponding to the icon 111.

[0211] In some embodiments, the above-mentioned start condition 2 includes: the mobile phone 100 does not meet the start condition 1. In one implementation, the start condition 2 includes at least one of the following: in the anti-mistouch scene, the anti-mistouch function is turned on, the screen is not facing the user, not inclined to be upright, and the distance between the mobile phone 100 and the computer 200 is greater than or equal to distance 2.

[0212] In some embodiments, the above-mentioned start condition 2 includes: the mobile phone 100 does not meet the start condition 1, and the page running in the foreground of the terminal device 100 belongs to the application 1 that cannot be applied to the computer 200. The application 1 can be a system application (such as the desktop, the negative one screen, the control bar, etc.) or a third-party application. The mobile phone 100 has version 1 of the application 1 installed (i.e., the mobile phone version of the application 1); the computer 200 cannot apply the application 1 to continue, which means: there is no version 2 of the application 1 applicable to the device type of the computer 200 (i.e., the computer version of the application 1); or, there is a version 2 of the application 1 applicable to the computer 200, but the computer 200 does not install the application 1 of this version. It can be understood that the version 1 applicable to the mobile phone and the version 2 applicable to the computer are different versions; if the aforementioned terminal device 100 and the terminal device 200 are of the same device type (such as a mobile phone), the versions of the application 1 applicable to the two devices can be the same.

[0213] In some embodiments, the computer 200 instructs the mobile phone 100 to project the preset screen display content to the computer 200, including: the computer 200 sends a message to the mobile phone 100, the message is used to instruct the mobile phone 100 to project the screen to the computer 200; the mobile phone 100 creates a virtual display screen for projection, runs the user interface (such as the main interface 13) to be projected to the computer 200 on the virtual display screen, and sends the image data of the display content of the virtual display screen to the computer 200; the computer 200 synchronously displays the display content of the virtual display screen in the projection window according to the above image data. When the multi-mode input subsystem of the computer 200 detects an input event in the projection window, the input event is converted into an input event of the virtual display screen of the mobile phone 100, and sent to the mobile phone 100; the multi-mode input subsystem of the mobile phone 100 can monitor the above input event transmitted by the computer 200, and process and distribute the above input event, and then the mobile phone 100 updates the display content of the virtual display screen according to the above input event, and sends the updated display content to the computer 200; the computer 200 will display the updated display content in the projection window. The embodiments of the present application do not specifically limit the implementation method of screen projection.

[0214] In some embodiments, the computer 200 receives and responds to a preset operation 1. If it is detected that the device status of the mobile phone 100 satisfies the start condition 2, the computer 200 performs an application connection for the application running in the foreground of the mobile phone 100.

[0215] For example, Figure 4GAs shown, the device status of the mobile phone 100 meets the start condition 3, and the mobile phone 100 runs the chat page 16 of the contact "Henry" in the instant messaging application in the foreground. The mobile phone 100 instructs the computer 200 to perform application continuation of the instant messaging application and sends the real-time usage data of the instant messaging application to the computer 200. The real-time usage data can indicate the page logo of the chat page 16 and the user's browsing position in the chat page 16; according to the above real-time usage data, the computer 200 runs the instant messaging application, and displays the homepage 18 of the instant messaging application in area 1 of the application continuation window 17, and displays the chat page 19 of the contact "Henry" in area 2 according to the above browsing position. The computer 200 can manipulate the display content in the application continuation window. For example, Figure 4G and Figure 4H As shown, the home page 18 includes an option 112 for "Group Chat 1"; after the cursor moves to option 112, it receives and responds to an operation on option 112 (such as a left-click operation performed by the user using a mouse), and the computer 200 displays the chat page 20 of the contact's "Group Chat 1" in area 2 of the application connection window 17.

[0216] In some embodiments, the above-mentioned start condition 2 includes: the mobile phone 100 does not meet the start condition 1.

[0217] In some embodiments, the above-mentioned start condition 2 includes: the mobile phone 100 does not meet the start condition 1, and the page running in the foreground of the terminal device 100 belongs to application 2 that the computer 200 can apply to continue.

[0218] In some embodiments, the mobile phone 100 has version 1 of application 2 installed (i.e., mobile phone version of application 2); the computer 200 can apply the subsequent application 2, which means: there is a version 2 of application 2 suitable for the computer 200, and the computer 200 has this version of application 2 installed.

[0219] In some embodiments, reference Fig. 4I If it is detected that there is a version 2 of application 2 applicable to computer 200, but computer 200 has not installed the version of application 2, computer 200 may display prompt information 113, which is used to prompt whether to install version 2 of application 2, and may also display an installation control 114 and a cancel control 115 of version 2 of application 2. The installation control 114 is used to trigger the installation of version 2 of application 2; the cancel control 115 is used to cancel the installation of version 2 of application 2. Fig. 4I As shown, if the computer 200 detects that the user has successfully installed application 2 by installing control 114, the computer 200 will continue the application 2 running in the foreground of the mobile phone 100; if the computer 200 detects an operation to cancel the control 115 (for example, a left-click operation performed by a mouse), the computer 200 instructs the mobile phone 100 to cast the screen to the computer 200.

[0220] In some embodiments, the computer 200 receives and responds to the preset operation 1. If it is detected that the device status of the mobile phone 100 meets the start condition 2, the computer 200 instructs the mobile phone 100 to perform multi-screen collaboration. If the page running in the foreground of the mobile phone 100 is an application that cannot be applied to continue, the multi-screen collaboration refers to the mobile phone 100 casting the screen to the computer 200; otherwise, the multi-screen collaboration refers to the computer 200 performing application continuation on the application running in the foreground of the mobile phone 100.

[0221] For example, see Figure 5A and Figure 5B , the mobile phone 100 may also be provided with a hyperterminal page. Exemplarily, Figure 5A The control center interface 21 of the mobile phone 100 provided in the embodiment of the present application includes a card 116 of a hyperterminal; after receiving and responding to an operation on the card 116, the mobile phone 100 can display a hyperterminal page 22. The hyperterminal page 22 includes an icon 117 of the mobile phone 100 and at least one terminal device that can establish a collaborative connection with the mobile phone 100, such as an icon 118 of a computer 200. For the specific implementation of the hyperterminal page 22, reference can be made to the relevant description of the hyperterminal page 12.

[0222] In one implementation, the mobile phone 100 receives and responds to a preset operation 1 for the icon 118 of the mobile phone 100 in the super terminal page 12 (for example, the user drags the icon 118 of the computer 200 close to the icon 117 of the mobile phone 100), and the mobile phone 100 requests to establish a collaborative connection with the computer 200 and detects the device status of the mobile phone 100; when it is detected that the device status of the mobile phone 100 meets the startup condition, the mobile phone 100 automatically instructs the computer 200 to perform the collaborative mode corresponding to the startup condition.

[0223] In some embodiments, mobile phone 100 receives and responds to preset operation 1. If mobile phone 100 detects that the device status of mobile phone 100 meets start condition 1, it sends a message to computer 200, and the message is used to instruct computer 200 to share the input device with mobile phone 100; if mobile phone 100 detects that the device status of mobile phone 100 meets start condition 2, it instructs computer 200 to perform multi-screen collaboration.

[0224] Not limited to the super terminal page, the embodiments of the present application can also trigger the mobile phone 100 / computer 200 to detect the startup conditions met by the device status of the mobile phone 100 in other ways, and automatically perform the collaborative mode corresponding to the startup conditions, which is not specifically limited here.

[0225] Interface Example 2

[0226] In the second embodiment of the interface, when the collaborative mode is initially started, the user can trigger the mobile phone 100 and the computer 200 to enter different collaborative modes through different user operations. The above collaborative modes may include one or more of the following: input device sharing, screen projection, and application continuation. The embodiment of the present application does not specifically limit the triggering method of the mobile phone 100 / computer 200 to trigger input device sharing, screen projection, and application continuation.

[0227] For example, FIG. 6A to FIG. 6G It shows that the user triggers the two devices to enter different collaborative modes through different input operations.

[0228] In some embodiments, the mobile phone 100 receives a response to a preset operation 2 of the user, instructing the computer 200 to share the input device with the mobile phone 100 .

[0229] In some embodiments, the computer 200 receives a response to the user's preset operation 2, and the computer 200 shares the input device with the mobile phone 100. Exemplarily, Fig. 6A and Figure 6B It shows an interface implementation in which a user shares an input device with a mobile phone 100 via a computer 200 .

[0230] like Fig. 6A As shown, the user opens the input device sharing setting page 31 on the computer 200. The setting page 31 displays a list of terminal devices that can be shared and a list of input devices that can be shared. The user can select one or more terminal devices from the terminal device list as target devices, and select one or more input devices from the input device list to share with the target devices. Fig. 6A As shown, the terminal device list may include one or more device columns of terminal devices that can be shared, such as the device column 201 of the mobile phone 100. The device column 201 of the mobile phone 100 may include a selection control 201A, a device identifier 201B of the mobile phone 100, a connection status 201C, and a control control 201D. The connection status 201C is used to indicate the current connection status of the mobile phone 100 and the computer 200, and the connection status may be one of unconnected, connected, and unpaired; the control control 201D is used to disconnect when the connection status is connected, trigger connection when the connection status is unconnected, and trigger pairing and connection when the connection status is unpaired. The input device list may include one or more device columns of input devices that can be shared, such as the device column 202 of the mouse. The device column 202 of the mouse may include a device identifier 202A and a selection control 202B of the mouse.

[0231] like Figure 6BAs shown, the user selects the mobile phone 100 by selecting the control 201A, and selects the mouse by selecting the control 202B. The setting page 31 also includes the completion control 203; the computer 200 receives and responds to the user's input operation on the completion control 203, and shares the mouse with the mobile phone 100.

[0232] In some embodiments, the mobile phone 100 receives a preset operation 3 in response to the user and projects the preset screen display content to the computer 200 .

[0233] In some embodiments, the computer 200 receives a preset operation 3 in response to the user, instructing the mobile phone 100 to project the preset screen display content to the computer 200. The preset operation 3 is different from the preset operation 2. For example, FIG. 6C to FIG. 6E It shows an interface implementation in which a user triggers the projection of the screen of the mobile phone 100 to the computer 200 through the mobile phone 100.

[0234] like Figure 6C As shown, when the user brings the NFC area of ​​the mobile phone 100 close to the NFC area of ​​the computer 200, the NFC connection between the mobile phone 100 and the computer 200 can be triggered, and the desktop of the mobile phone 100 can be projected to the computer 200. In one implementation, after the user brings the NFC area of ​​the mobile phone 100 close to the NFC area of ​​the computer 200, the mobile phone 100 can detect the NFC signal of the computer 200 and display it on the current display interface. Fig.6D The prompt box 204 shown includes prompt information 204A, a connection control 204B, and a cancel control 204C. The prompt information 204A is used to prompt the user to click the connection control 204B to trigger the screen projection to the computer 200; the cancel control 204C is used to close the prompt box 204.

[0235] like Fig.6D and Fig. 6E As shown, the mobile phone 100 receives and responds to the input operation (such as touch operation) for the connection control 204B, and sends the projection data to the computer 200; the computer 200 displays the projection window 15 corresponding to the mobile phone 100 on the current display interface (such as the main interface 11 of the computer 200) based on the above projection data, and the projection window 15 displays the main interface 13 from the mobile phone 100.

[0236] In some embodiments, the mobile phone 100 receives a response to a preset operation 4 of the user, instructing the computer 200 to perform an application connection for the application 2 currently running on the mobile phone 100 .

[0237] In some embodiments, the computer 200 receives a preset operation 4 in response to the user, and the computer 200 performs an application connection to the application 2 currently running on the mobile phone 100. The preset operation 4 is different from the preset operation 3 and the preset operation 2. For example, FIG. 6F to FIG. 6GIt shows an interface implementation in which a user triggers the computer 200 to connect the application 2 of the mobile phone 100 through the computer 200 .

[0238] In an embodiment of the present application, when a user opens a new application, the computer 200 can switch the application running in the foreground to the background, add the application window of the running application to the multi-task queue as a historical task, and store a snapshot of the user interface last displayed in the above application window as an interface snapshot of the historical task.

[0239] Fig. 6F The figure shows a historical task interface 32 on the computer 200. The historical task interface 32 includes an icon 205 of the computer 200 and an icon of a terminal device that has established a collaborative connection with the computer 200, such as an icon 206 of the mobile phone 100. The historical task interface 32 can be used to display the historical tasks of the computer 200 and the historical tasks of the mobile phone 100. Fig. 6F As shown, the icon 205 of the computer 200 is in a selected state. At this time, the historical task interface 32 displays the task cards of the historical tasks of each application in the multi-task queue of the computer 200. Fig. 6F and Figure 6G As shown, in response to receiving and inputting an operation on the icon 206 of the mobile phone 100, the computer 200 switches the icon 206 of the mobile phone 100 from an unselected state to a selected state, and displays task cards of historical tasks of each application in the multi-task queue of the mobile phone 100 on the historical task interface 32, such as the task card 207 of the instant messaging application, and the task card 207 displays an interface snapshot of the chat page 16 of the contact "Henry".

[0240] In some embodiments, Figure 6G and Figure 6H As shown, upon receiving and responding to an input operation on a task card 207 of an instant messaging application, the computer 200 performs an application connection on the instant messaging application and displays a chat page 19 of the contact “Henry” corresponding to the task card 207 in the application connection window 17 .

[0241] Interface Example 3

[0242] In the third embodiment of the interface, the mobile phone 100 and the computer 200 can automatically switch the collaborative mode according to the device status of the mobile phone 100. Specifically, when the mobile phone 100 and the computer 200 have currently started the first collaborative mode, if the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the start-up conditions corresponding to the second collaborative mode, the mobile phone 100 / computer 200 automatically switches the first collaborative mode to the second collaborative mode, that is, stops the first collaborative mode and starts the second collaborative mode. For example, the first collaborative mode is input device sharing, and the second collaborative mode is multi-screen collaboration (such as screen projection, application continuation); or, the first collaborative mode is multi-screen collaboration, and the second collaborative mode is input device sharing.

[0243] In some embodiments, the computer 200 currently shares the input device with the mobile phone 100. If the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the start condition 2, the input device sharing is automatically switched to screen projection, that is, the computer 200 stops sharing the input device with the mobile phone 100, and the mobile phone 100 projects the screen to the computer 200.

[0244] In some embodiments, mobile phone 100 is currently projecting the screen to computer 200. If mobile phone 100 / computer 200 detects that the device status of mobile phone 100 meets start condition 1, the screen projection is automatically switched to input device sharing, that is, mobile phone 100 stops projecting the screen to computer 200, and computer 200 shares the input device with mobile phone 100.

[0245] In some embodiments, the computer 200 currently shares the input device with the mobile phone 100. If the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the start condition 2, the input device sharing is automatically switched to application continuation, that is, the computer 200 stops sharing the input device with the mobile phone 100, and performs application continuation for the application 2 currently running on the mobile phone 100.

[0246] In some embodiments, the computer 200 performs an application connection for the application 2 running on the mobile phone 100. If the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the start condition 1, the application connection is automatically switched to input device sharing, that is, the computer 200 closes the application connection window, and the computer 200 shares the input device with the mobile phone 100.

[0247] In some embodiments, the computer 200 currently shares the input device with the mobile phone 100. If the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the start condition 2, the input device sharing is switched to multi-screen collaboration (e.g., screen projection, application continuation). In some embodiments, the mobile phone 100 and the computer 200 are currently performing multi-screen collaboration. If the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the start condition 1, the multi-screen collaboration is switched to input device sharing. Among them, the above-mentioned switching of input device sharing to multi-screen collaboration includes: if the page running in the foreground of the terminal device 100 belongs to an application that cannot be applied in succession, the multi-screen collaboration refers to the mobile phone 100 casting the preset screen display content to the computer 200; otherwise, the multi-screen collaboration refers to the computer 200 performing application continuation on the application running in the foreground of the mobile phone 100.

[0248] For the above-mentioned start condition 1 and start condition 2, reference may be made to the relevant description of the first interface embodiment.

[0249] In one implementation, the start condition 1 includes any of the following: the mobile phone 100 is not in the accidental touch prevention scene, the mobile phone 100 does not turn on the accidental touch prevention function, the screen of the mobile phone 100 faces the user, the mobile phone 100 tends to be upright, and the distance between the mobile phone 100 and the computer 200 is less than the distance 2. The start condition 2 includes: the device state of the mobile phone 100 does not meet the start condition 1, and the page running in the foreground of the mobile phone 100 is an application that cannot be applied to continue. The start condition 3 includes: the device state of the mobile phone 100 does not meet the start condition 1, and the page running in the foreground of the mobile phone 100 is an application that can be applied to continue.

[0250] For example, Figure 7 As shown in (a), the mobile phone 100 is in an upright posture, or / and the mobile phone 100 is facing the user, the device state of the mobile phone 100 meets the start condition 1, and the computer 200 currently shares the input device (such as a mouse) with the mobile phone 100.

[0251] like Figure 7 As shown in (b), when the mobile phone 100 detects that the mobile phone 100 is placed flat and / or is not facing the user, and an application (such as the main interface 13) that cannot be connected is running in the foreground of the mobile phone 100, the mobile phone 100 stops the input event of the input device of the computer 200, instructs the computer 200 to stop sharing the mouse with the mobile phone 100, and casts the currently displayed user interface (such as the main interface 13) to the computer 200, and the computer 200 displays the user interface from the mobile phone 100 in the projection window 15. On the contrary, Figure 7After (b), if the user switches the mobile phone 100 from a flat position back to an upright position, and / or switches the screen from not facing the user to facing the user, the mobile phone 100 can stop screen projection when it detects that the device status of the mobile phone 100 meets the start condition 1, and instruct the computer 200 to switch the screen projection back to input device sharing.

[0252] like Figure 7 As shown in (c), the mobile phone 100 displays the chat page 16 of the contact "Henry" in the instant messaging application. When the mobile phone 100 detects that the mobile phone 100 is placed flat and / or is not facing the user, and the mobile phone 100 is running an instant messaging application that can be connected to the application in the foreground, the mobile phone 100 instructs the computer 200 to stop sharing the input device with the mobile phone 100, and instructs the computer 200 to connect the instant messaging application of the mobile phone 100, and displays the chat page 19 of the contact "Henry" in the instant messaging application in the application connection window 17. On the contrary, Figure 7 After (c), if the user switches the mobile phone 100 from a flat position back to an upright position, and / or switches the screen from not facing the user back to facing the user, when the mobile phone 100 detects that the device status of the mobile phone 100 meets the start condition 1, it can instruct the computer 200 to switch the application back to input device sharing.

[0253] For example, Figure 8 As shown in (a), the distance between the mobile phone 100 and the computer 200 is less than distance 2, the current device state of the mobile phone 100 meets the start condition 1, and the computer 200 currently shares the input device (such as a mouse) with the mobile phone 100.

[0254] like Figure 8 As shown in (b), the user moves the mobile phone 100 away from the computer 200, and the mobile phone 100 detects that the distance between the mobile phone 100 and the computer 200 is greater than or equal to distance 2, and the mobile phone 100 runs a system application that cannot be connected in the foreground, the mobile phone 100 instructs the computer 200 to stop sharing the input device with the mobile phone 100, and casts the currently displayed main interface 13 to the computer 200, and the computer 200 displays the main interface 13 from the mobile phone 100 in the projection window 15. On the contrary, Figure 8 After (b), if the user moves the mobile phone 100 closer to the computer 200 so that the distance between the mobile phone 100 and the computer 200 is less than distance 2, when the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the start condition 1, it can instruct the computer 200 to switch the screen projection back to input device sharing.

[0255] like Figure 8As shown in (c), the user moves the mobile phone 100 away from the computer 200, and the mobile phone 100 displays the chat page 16 of the contact "Henry" in the instant messaging application. When the mobile phone 100 detects that the distance between the mobile phone 100 and the computer 200 is greater than or equal to the distance 2, and the mobile phone 100 runs an instant messaging application that can be connected to the application in the foreground, the mobile phone 100 instructs the computer 200 to stop sharing the input device with the mobile phone 100, and instructs the computer 200 to connect the instant messaging application of the mobile phone 100, and displays the chat page 19 of the contact "Henry" in the instant messaging application in the application connection window 17. On the contrary, Figure 8 After (c), if the user moves the mobile phone 100 close to the computer 200 so that the distance between the mobile phone 100 and the computer 200 is less than distance 2, the mobile phone 100 / computer 200 detects that the device status of the mobile phone 100 meets the start condition 1, and can instruct the computer 200 to switch the application back to input device sharing.

[0256] Interface Example 4

[0257] In the fourth embodiment of the interface, the user can manually switch the collaborative mode. Specifically, when the mobile phone 100 and the computer 200 are in the first collaborative mode, the user can trigger the mobile phone 100 and the computer 200 to switch the first collaborative mode to the second collaborative mode by acting on the preset operation of the mobile phone 100 / computer 200, that is, stop the first collaborative mode and start the second collaborative mode. For example, the first collaborative mode is input device sharing, and the second collaborative mode is multi-screen collaboration (such as screen projection, application continuation); or, the first collaborative mode is multi-screen collaboration, and the second collaborative mode is input device sharing.

[0258] In some embodiments, the computer 200 currently shares the input device with the mobile phone 100, and the mobile phone 100 / computer 200 receives and responds to the user's preset operation 5, and controls the computer 200 and the mobile phone 100 to switch the input device sharing to screen projection. In some embodiments, the mobile phone 100 currently projects the screen to the computer 200, and the mobile phone 100 / computer 200 receives and responds to the user's preset operation 6, and controls the computer 200 and the mobile phone 100 to switch the screen projection to input device sharing.

[0259] In some embodiments, the computer 200 currently shares the input device with the mobile phone 100, and the mobile phone 100 / computer 200 receives and responds to the user's preset operation 5. When the page running in the foreground of the mobile phone 100 belongs to an application that cannot be applied to the continuation, the computer 200 and the mobile phone 100 are controlled to switch the input device sharing to screen projection. When the page running in the foreground of the mobile phone 100 belongs to an application that can be applied to the continuation, the computer 200 and the mobile phone 100 are controlled to switch the input device sharing to application continuation. In some embodiments, the mobile phone 100 currently projects the screen to the computer 200, and the mobile phone 100 / computer 200 receives and responds to the user's preset operation 6, and controls the computer 200 and the mobile phone 100 to switch the screen projection to input device sharing. In some embodiments, the computer 200 currently performs application continuation on the application 2 of the mobile phone 100, and the mobile phone 100 / computer 200 receives and responds to the user's preset operation 7, and controls the computer 200 and the mobile phone 100 to switch the application continuation to input device sharing.

[0260] The embodiments of the present application do not specifically limit preset operation 5, preset operation 6, and preset operation 7.

[0261] For example, 9A to 9D An interface implementation for manually switching collaborative modes is shown.

[0262] In the embodiment of the present application, after the computer 200 shares the input device with the mobile phone 100, the user can use the input device of the computer 200 (such as a mouse) to drag the pictures, videos, folders, files (such as word documents), user interfaces corresponding to the application windows, and other contents displayed on the mobile phone 100 to the computer 200 for display and operation.

[0263] In some embodiments, Fig.9A As shown, after the computer 200 shares the input device with the mobile phone 100, the user can use the input device of the computer 200 (such as a mouse) to drag the user interface currently displayed by the mobile phone 100 (such as the main interface 13) to the screen of the computer 200, thereby triggering the computer 200 and the mobile phone 100 to switch the input device sharing to screen projection.

[0264] In one implementation, Fig.9AAs shown, after the cursor corresponding to the mouse moves to the screen of the mobile phone 100, when the mobile phone 100 detects that the user selects the main interface 13 with the mouse (for example, by long pressing the left button of the mouse on a blank area of ​​the main interface 13), the interface snapshot 301 of the main interface 13 is acquired and displayed; then, when it is detected that the user moves the mouse, the cursor and the interface snapshot 301 are controlled to move with the mouse, and the relative positions of the cursor and the interface snapshot 301 remain unchanged, presenting a visual effect of dragging the main interface 13; as the mouse continues to move, if the cursor moves to the mobile phone 10 0 and then continues to move toward the outside of the screen, the cursor dragging interface snapshot 301 penetrates into the left border of the screen of the computer 200; the computer 200 detects that the cursor dragging interface snapshot 301 has reached the screen of the computer 200 and the user releases the mouse (i.e. stops pressing the left button of the mouse), stops sharing the mouse with the mobile phone 100, and instructs the mobile phone 100 to project the screen to the computer 200; the mobile phone projects the currently displayed main interface 13 to the computer 200, and the computer 200 displays the main interface 13 from the mobile phone 100 in the projection window 15.

[0265] In another implementation, the computer 200 switches the input device sharing to screen casting after detecting that the cursor dragging interface snapshot 301 moves to the screen of the computer 200. In another implementation, the mobile phone 100 instructs the computer 200 to switch the input device sharing to screen casting when detecting that the cursor dragging interface snapshot 301 moves out of the screen of the mobile phone 100.

[0266] In the embodiment of the present application, the mobile phone 100 may send the interface snapshot 301 to the computer 200 after acquiring the interface snapshot 301, or may send the interface snapshot 301 to the computer 200 when the cursor crosses the screen boundary of the mobile phone 100, which is not specifically limited here.

[0267] In some embodiments, reference Fig.9A After the computer 200 shares the input device (such as a mouse) with the mobile phone 100, the user can use the mouse to drag the user interface (such as the main interface 13) currently displayed on the mobile phone 100 to the screen of the computer 200. When it is determined that the dragged user interface is an application that cannot be applied to the continuation, the computer 200 switches the input device sharing to screen projection. In one implementation, along with the interface snapshot 301, the mobile phone 100 also sends an identifier 1 to the computer 200, and the identifier 1 is used to indicate that the main interface 13 corresponding to the interface snapshot 301 does not belong to an application that can be applied to the continuation.

[0268] In some embodiments, when the cursor is dragging the interface snapshot 301, the interface snapshot 301 moves out of the area of ​​the mobile phone screen and is moved into the screen of the computer 200. That is, while the computer 200 displays the first area of ​​the interface snapshot 301 that moves into the computer screen, the mobile phone 100 displays the second area outside the first area of ​​the interface snapshot 301.

[0269] For example, Fig.9A As shown, the distance between the cursor and the right border of the interface snapshot is a first value, and the distance between the cursor and the left border of the interface snapshot is a second value. In one implementation, when the cursor drags the interface snapshot 301 to move on the mobile phone screen, when the distance between the cursor and the right border of the mobile phone screen is less than the first value and continues to decrease, the first area of ​​the interface snapshot 301 moves out of the mobile phone screen and moves into the computer screen; while the mobile phone 100 displays the first area of ​​the interface snapshot 301, the computer 200 displays the second area outside the first area in the interface snapshot 301. The first area and the second area are divided by the right border of the mobile phone screen, and the second area is the left half of the interface snapshot 301. After the cursor drags the interface snapshot 301 to the computer screen, the greater the distance between the cursor and the left border of the computer screen, the larger the first area moved into the computer screen; when the distance between the cursor and the left border of the computer screen is less than the second value, while the computer 200 displays the first area of ​​the interface snapshot 301, the mobile phone 100 displays the second area in the interface snapshot 301. Similarly, in subsequent embodiments, when the cursor is dragging a window, an interface snapshot, or other object to move, the computer 200 and the mobile phone 100 may respectively display a portion of the dragged object, which will not be described in detail in the subsequent embodiments.

[0270] In some embodiments, Fig.9A As shown, when the cursor drags the interface snapshot 301 to move on the mobile phone screen, the computer 200 does not display the interface snapshot 301; when the cursor drags the interface snapshot 301 to the computer screen, the mobile phone 100 stops displaying the interface snapshot 301. Similarly, in the subsequent embodiments, when the cursor drags the window, interface snapshot, etc., the computer 200 and the mobile phone 100 cannot simultaneously display part of the dragged object, and the subsequent embodiments will not be repeated.

[0271] like Fig. 9B As shown, after the computer 200 displays the projection window 15 of the mobile phone 100, the user can use the mouse to drag the projection window 15 back to the screen of the mobile phone 100, thereby triggering the mobile phone 100 and the computer 200 to switch the projection to input device sharing.

[0272] In one implementation, Fig. 9BAs shown, the projection window 15 displays the chat page 16; the computer 200 detects that the user selects the projection window 15 with the mouse (for example, by long pressing the left button of the mouse on the blank area of ​​the projection window 15); when it is detected that the user moves the mouse, the cursor and the projection window 15 are controlled to move with the mouse, and the relative positions of the cursor and the projection window 15 remain unchanged, presenting a visual effect of the cursor dragging the projection window 15 to move; as the mouse continues to move, the computer 200 detects that the cursor moves to the left border of the computer 200 screen, moves to the left border of the computer 200 screen and stays at the left border for longer than a preset value, or moves to the left of the computer 200 screen. When the cursor continues to move toward the outside of the screen after reaching the left border of the screen, an interface snapshot 302 is sent to the mobile phone 100, instructing the mobile phone 100 to stop projecting the screen to the computer 200, and sharing the input device (such as a mouse) with the mobile phone 100; after the mouse is shared with the mobile phone 100, as the mouse continues to move, the mobile phone 100 displays the cursor dragging interface snapshot 302 penetrating the right border of the screen of the mobile phone 100; after the mobile phone 100 detects that the cursor drags the interface snapshot 302 to the screen of the mobile phone 100, and the user releases the mouse, the interface snapshot 302 of the projection window 15 is stopped from being displayed, and the chat page 16 in the projection window 15 is run and displayed in the foreground. Reference Fig. 9B When the user drags the projection window 15 of the computer 200 back to the screen of the mobile phone 100, the user interface displayed in the projection window 15 (such as the main interface 13) and the user interface displayed in the mobile phone 100 (such as the chat page 16) may be different or the same.

[0273] Among them, the computer 200 can obtain an interface snapshot 302 of the projection window 15 when it detects that the user selects the projection window 15, or it can obtain an interface snapshot 302 when it detects that the cursor moves to the left border of the computer 200 screen and continues to move to the outside of the screen. No specific limitation is made here.

[0274] In some embodiments, reference Fig. 9C After the computer 200 shares the input device with the mobile phone 100, the user can use the input device of the computer 200 (such as a mouse) to drag the user interface currently displayed by the mobile phone 100 (such as the chat page 16) to the screen of the computer 200, thereby triggering the computer 200 to switch the input device sharing to application continuation.

[0275] In one implementation, Fig. 9CAs shown, when the mobile phone 100 detects that the user selects the chat page 16 with the mouse, it obtains and displays the interface snapshot 303 of the chat page 16; when it detects that the user moves the mouse, it controls the cursor and the interface snapshot 303 to move with the mouse; if the cursor moves to the right boundary of the screen of the mobile phone 100 and continues to move to the outside of the screen, the computer 200 displays the cursor dragging the interface snapshot 303 to penetrate the left boundary of the screen of the computer 200; after the computer 200 detects that the cursor drags the interface snapshot 303 to the screen of the computer 200 and the user releases the mouse, it stops sharing the mouse with the mobile phone 100, and performs application continuation on the chat page 16 of the instant messaging application of the mobile phone 100, and displays the application continuation window 17.

[0276] In one implementation, along with the interface snapshot 303, the mobile phone 100 can also send an identifier 2 to the computer 200, and the identifier 2 is used to indicate that the chat page 16 corresponding to the interface snapshot 303 belongs to an application that can be applied to continue. The interface snapshot 303 can refer to the relevant description of the interface snapshot 301, which will not be repeated here.

[0277] In some embodiments, Fig.9D As shown, after the computer 200 displays the application connection window 17 of the instant messaging application, the user can use the mouse to drag the application connection window 17 to the screen of the mobile phone 100, thereby triggering the computer 200 to switch the application connection to input device sharing.

[0278] In one implementation, Fig.9D As shown, the computer 200 displays the application continuation window 17; when the computer 200 detects that the user selects the application continuation window 17 with the mouse and moves the mouse, the computer 200 controls the cursor and the application continuation window 17 to move with the mouse; if the cursor moves to the left border of the computer 200 screen, moves to the left border of the computer 200 screen and stays at the left border for longer than a preset value, or moves to the left border of the computer 200 screen and continues to move to the outside of the screen, the computer 200 closes the application continuation window 17 in the background and shares the input device (such as the mouse) with the mobile phone 100; after the mouse is shared with the mobile phone 100, as the mouse continues to move, the mobile phone 100 displays the interface snapshot 304 of the cursor dragging the application continuation window 17 to penetrate the right border of the screen of the mobile phone 100; as shown Fig.9D As shown, the mobile phone 100 detects that the cursor drags the interface snapshot 304 to the screen of the mobile phone 100, and after the user releases the mouse, the interface snapshot 304 of the application continuation window 17 stops displaying. In one implementation, the application continuation window 17 displays the chat page 19 of the contact "Henry" of the computer version of the instant messaging application. After the mobile phone 100 stops displaying the interface snapshot 304 of the application continuation window 17, the chat page 16 of "Henry" of the mobile version of the instant messaging application runs in the foreground.

[0279] like Fig.9D As shown, when the application continuation window 17 of the computer 200 is dragged back to the screen of the mobile phone 100, the application displayed in the application continuation window 17 (such as an instant messaging application) and the application displayed in the mobile phone 100 (such as a desktop application) can be different or the same.

[0280] For example, 9A to 9D The method of manually switching the collaborative mode described in Fig.9E A software implementation process of manually switching the collaborative mode is shown. Figure 3 and Fig.9E As shown, the UI framework may include system controls; the basic capability subsystem set of the system service layer may also include a drag framework.

[0281] In some embodiments, Fig.9F As shown, when the computer 200 shares the input device with the mobile phone 100, the user can use the input device (such as a mouse, a touch screen pad) to drag the interface snapshot of the user interface 1 (such as the main interface 13, the chat page 16) currently displayed on the mobile phone 100 to the screen of the computer 200, thereby triggering the computer 200 to switch the input device sharing to multi-screen collaboration (such as screen projection, application continuation). For example, see Fig.9E , Fig.9A and Fig. 9CAfter the computer 200 shares the input device with the mobile phone 100, when the mouse cursor moves to the screen of the mobile phone 100, the multi-mode input subsystem of the mobile phone 100 can monitor the input event of the input device (such as the mouse) shared by the computer 200 for the mobile phone 100; for example, when the user selects the user interface currently displayed by the mobile phone 100 by pressing and holding the left button of the mouse, the multi-mode input subsystem detects the mouse input event 1 and sends the input event 1 to the system control of the UI framework; the system control determines that the input event 1 is used to select the user interface 1 according to the interface layout of the user interface running in the foreground, and the system control instructs the drag framework to obtain the interface snapshot of the user interface 1, and the drag framework can share the interface snapshot with the computer 200 through the distributed data management module; the mobile phone 100 displays the mouse cursor and the above-mentioned interface snapshot; when the user drags the above-mentioned interface snapshot to move with the mouse, the multi-mode input subsystem detects the mouse input event 2 and sends the input event 2 to the traversal service; the traversal service determines the mouse cursor position according to the input event 2, and instructs the drag framework to display the cursor dragging interface snapshot movement according to the above-mentioned cursor position; the traversal service detects that the cursor moves to the mobile phone 1 00 and continues to move toward the outside of the screen, the drag frame is instructed to stop displaying the cursor drag interface snapshot and send a crossing information to the crossing service of the computer 200; the crossing service of the computer 200 informs the multi-mode input subsystem and the drag frame of the crossing information; the crossing information instructs the multi-mode input subsystem of the computer 200 to locally process the mouse input event; the drag frame of the computer 200 obtains the interface snapshot of the user interface 1 sent by the drag frame of the mobile phone 100, triggering the computer 200 to display the cursor and the interface snapshot; the user drags the interface snapshot on the computer 200 by the mouse When the cursor moves on the screen, the multi-mode input subsystem detects the mouse input event 4 and sends the input event 4 to the crossing service; the crossing service determines the cursor position based on the input event 4, and instructs the drag framework to display the cursor dragging interface snapshot based on the above cursor position; when the user stops dragging the interface snapshot with the mouse and releases the mouse, the multi-mode input subsystem detects the mouse input event 5 and sends the input event 5 to the system control of the UI framework. The system control of the UI framework triggers the mobile phone 100 to switch the input device sharing to multi-screen collaboration (such as screen projection, application continuation) based on the input event 5.

[0282] In some embodiments, Fig. 9B or Fig.9D As shown, when the computer 200 performs multi-screen collaboration (such as screen projection, application continuation) on the mobile phone 100, the user can use an input device (such as a mouse or touch screen pad) to drag the multi-screen collaboration window (such as a screen projection window, an application continuation window) currently displayed on the computer 200 back to the screen of the mobile phone 100, thereby triggering the mobile phone 100 to switch the multi-screen collaboration back to input device sharing. For example, see Fig.9E , Fig. 9B and Fig.9D , computer 200 displays the multi-screen collaborative window of mobile phone 100. After the user selects the multi-screen collaborative window by long pressing the left button of the mouse, as the mouse moves, the computer 200 displays the cursor dragging the multi-screen collaborative window on the screen; when the cursor moves to the boundary of computer 200 and continues to move to the outside of the screen, the multi-mode input subsystem of computer 200 detects the input event 6 of the mouse and sends the input event 6 to the crossing service; based on input event 6, the crossing service instructs the drag framework to obtain an interface snapshot of the multi-screen collaborative window, triggering the computer 200 to switch the multi-screen collaboration to input device sharing, and send crossing information to the crossing service of mobile phone 100, and the drag framework shares the interface snapshot with mobile phone 100. After completing the sharing of the input device, the traversal service of the mobile phone 100 informs the multi-mode input subsystem of the above traversal information, and instructs the multi-mode input subsystem of the mobile phone 100 to monitor the input event of the mouse; the multi-mode input subsystem of the mobile phone 100 detects the input event 7, and sends the input event 7 to the traversal service, and the traversal service instructs the drag framework to obtain the interface snapshot of the multi-screen collaborative window shared by the computer 200, and the drag framework triggers the mobile phone 100 to display the cursor to drag the interface snapshot; when the user stops dragging the interface snapshot with the mouse and releases the mouse, the multi-mode input subsystem of the mobile phone 100 detects the input event 8 of the mouse, and sends the input event 8 to the system control of the UI framework, and based on the input event 8, the system control triggers the mobile phone 100 to stop displaying the interface snapshot of the multi-screen collaborative window. In some embodiments, based on the input event 8, the system control also triggers the mobile phone 100 to continue running the user interface in the multi-screen collaborative window.

[0283] For example, FIG. 9F to FIG. 9I Another interface implementation of collaborative mode switching is shown.

[0284] In some embodiments, the mobile phone 100 and / or the computer 200 are provided with switch controls for each collaborative mode of the connected collaborative devices, such as a switch control for input device sharing, a switch control for screen projection, and a switch control for application continuation; a switch control for a collaborative mode is used to control the start / stop of the collaborative mode.

[0285] In some embodiments, at most one switch control of a collaborative mode can be in an on state, and a user can control switching of the collaborative mode through the switch control.

[0286] For example, Fig.9FAs shown, the computer 200 is provided with switch controls for each collaborative mode of the connected mobile phone 100. After the computer 200 establishes a collaborative connection with the mobile phone 100 through the hyperterminal, the status bar of the computer 200 displays an icon 401 of the connected mobile phone 100; upon receiving and responding to an input operation (such as a click operation) on the icon 401 of the mobile phone 100, the computer 200 displays the setting page 21 of the mobile phone 100, which displays a control 402, a switch control 403 for input device sharing, and a switch control 404 for multi-screen collaboration.

[0287] Among them, the control 402 is used to disconnect the collaborative connection with the mobile phone 100. When the switch control 403 for sharing the input device is switched to the on state (i.e., "ON"), the computer 200 shares the input device with the mobile phone 100; when the switch control 403 for sharing the input device is switched to the off state (i.e., "OFF"), the computer 200 stops sharing the input device with the mobile phone 100.

[0288] In one implementation, when the switch control 404 of multi-screen collaboration is switched to the on state (i.e., “ON”), the computer 200 instructs the mobile phone 100 to project the screen to the computer 200; when the switch control 404 of multi-screen collaboration is switched to the off state (i.e., “OFF”), the computer 200 instructs the mobile phone 100 to stop projecting the screen to the computer 200.

[0289] In another implementation, when the switch control 404 of multi-screen collaboration is switched to the on state (i.e., "ON"), the computer 200 instructs the mobile phone 100 to perform multi-screen collaboration. If the page running in the foreground of the mobile phone 100 is an application that cannot be applied and continued, the mobile phone 100 will cast the screen to the computer 200. If the page running in the foreground of the mobile phone 100 is an application that can be applied and continued, the computer 200 will perform application continuation on the page running in the foreground of the mobile phone 100. When the switch control 404 of multi-screen collaboration is switched to the off state (i.e., "OFF"), the computer 200 and the mobile phone 100 stop multi-screen collaboration (i.e., stop screen casting / application continuation).

[0290] In some embodiments, among the switch controls 403 for input device sharing and 404 for multi-screen collaboration, at most one switch control for collaboration mode can be in the on state, that is, input device sharing and multi-screen collaboration (i.e., screen projection or application continuation) cannot be turned on at the same time, and the switch control can be used to switch the collaboration mode.

[0291] For example, Fig.9FAs shown, when the computer 200 shares the input device with the mobile phone 100, the switch control 403 for input device sharing is in the on state, and the switch control 404 for multi-screen collaboration is in the off state; upon receiving and responding to an input operation (such as a click operation) for the switch control 404 for multi-screen collaboration, the computer 200 switches the switch control 403 for input device sharing to the off state, stops sharing the input device with the mobile phone 100, switches the switch control 403 for multi-screen collaboration to the on state, and instructs the mobile phone 100 to perform multi-screen collaboration; when the page running in the foreground of the mobile phone 100 is the main interface 13 that cannot be applied, the mobile phone 100 casts the screen to the computer 200, and the computer 200 displays the projection window 15, in which the main interface 13 of the mobile phone 100 is displayed.

[0292] For example, Figure 9G As shown, the mobile phone 100 is projected to the computer 200. When the computer 200 displays the projection window 15, the switch control 403 for input device sharing is in the off state, and the switch control 404 for multi-screen collaboration is in the on state; upon receiving and responding to an input operation (such as a click operation) for the switch control 403 for input device sharing, the computer 200 switches the switch control 403 for multi-screen collaboration to the off state, instructing the mobile phone 100 to stop projecting the screen, and switches the switch control 403 for input device sharing to the on state, sharing the input device with the mobile phone 100.

[0293] For example, Figure 9H As shown, when the computer 200 shares the input device with the mobile phone 100, the switch control 403 for input device sharing is in the on state, and the switch control 404 for multi-screen collaboration is in the off state; upon receiving and responding to an input operation (such as a click operation) for the switch control 404 for multi-screen collaboration, the computer 200 switches the switch control 403 for input device sharing to the off state, stops sharing the input device with the mobile phone 100, switches the switch control 403 for multi-screen collaboration to the on state, and instructs the mobile phone 100 to perform multi-screen collaboration; when the page running in the foreground of the mobile phone 100 is a chat page 16 that can be applied for continuation, the computer 200 performs application continuation on the chat page 16 of the instant messaging application, and displays the application continuation window 17 of the instant messaging application.

[0294] For example, Fig.9I As shown, when the computer 200 displays the application continuation window 17, the switch control 403 shared by the input device is in the off state, and the switch control 404 for multi-screen collaboration is in the on state; upon receiving and responding to an input operation (such as a click operation) for the switch control 403 shared by the input device, the computer 200 switches the switch control 403 for multi-screen collaboration to the off state, closes the application continuation window 17, and switches the switch control 403 shared by the input device to the on state, sharing the input device with the mobile phone 100.

[0295] In the embodiment of the present application, it is not limited to the setting page 21, and the switch controls of each collaborative mode can also be set on other pages of the computer 200, which is not specifically limited here. FIG. 9G to FIG. 9I In the example provided, screen projection and application connection can share a switch control; in other examples, screen projection and application connection can also correspond to a switch control respectively, which is not specifically limited here.

[0296] In other embodiments, when the mobile phone 100 and the computer 200 are in the first collaborative mode, if the mobile phone 100 / computer 200 detects that the mobile phone 100 meets the start-up conditions corresponding to the second collaborative mode, the mobile phone 100 and the computer 200 are automatically triggered to start the second collaborative mode, that is, the first collaborative mode and the second collaborative mode can be performed at the same time. In one implementation, the start-up conditions corresponding to the first collaborative mode and the start-up conditions corresponding to the second collaborative mode can coexist, and the developer can define the start-up conditions of each collaborative mode according to the usage requirements of the actual scenario. The embodiments of the present application do not specifically limit the start-up conditions of each collaborative mode. For example, the first collaborative mode is input device sharing, and the second collaborative mode is multi-screen collaboration; or, the first collaborative mode is multi-screen collaboration, and the second collaborative mode is input device sharing.

[0297] In some embodiments, the computer 200 currently shares the input device with the mobile phone 100. If it is detected that the device status of the mobile phone 100 meets the start condition of screen projection, the computer 200 will instruct the mobile phone 100 to project the screen to the computer 200 while sharing the input device with the mobile phone 100.

[0298] In some embodiments, the computer 200 currently shares the input device with the mobile phone 100. If it is detected that the device status of the mobile phone 100 meets the startup conditions for application continuation, the computer 200 will share the input device with the mobile phone 100 while performing application continuation for the application 2 running in the foreground of the mobile phone 100.

[0299] In some embodiments, the mobile phone 100 is currently projecting the screen to the computer 200. If it is detected that the device status of the mobile phone 100 meets the start conditions for sharing the input device, the computer 200 shares the input device with the mobile phone 100 while displaying the projection window corresponding to the mobile phone 100.

[0300] In some embodiments, when the computer 200 continues the application 1 of the mobile phone 100, if it detects that the device status of the mobile phone 100 meets the start condition of input device sharing, the computer 200 will share the input device with the mobile phone 100 while continuing the application 2 of the mobile phone 100.

[0301] In other embodiments, when the mobile phone 100 and the computer 200 are in the first collaborative mode, the user can manually trigger the mobile phone 100 and the computer 200 to start the second collaborative mode, that is, the first collaborative mode and the second collaborative mode can be performed simultaneously. Specifically, when the mobile phone 100 and the computer 200 are in the first collaborative mode, the user can trigger the mobile phone 100 and the computer 200 to start the second collaborative mode by acting on the preset operation of the mobile phone 100 / computer 200 (such as the preset operation 5, preset operation 6 and preset operation 7 mentioned above).

[0302] The present application provides a multi-device collaboration method, which is applied to a multi-device collaboration system, wherein the multi-device collaboration system includes a first terminal and a second terminal, and the second terminal is connected to a first input device. The method includes:

[0303] S101. A first terminal establishes a communication connection with a second terminal; the first terminal displays a first user interface.

[0304] In the embodiment of the present application, the first terminal may be the aforementioned terminal device 100, and the second terminal may be the aforementioned terminal device 200.

[0305] S102: The second terminal shares at least one input device with the first terminal, the first terminal and the second terminal are in an input device sharing mode, and the at least one input device includes the first input device;

[0306] S103: In response to a first operation of the first input device, the first terminal displays that a cursor of the first input device drags a first interface snapshot of the first user interface to move;

[0307] S104. After the cursor drags the snapshot of the first interface to the display screen of the second terminal, the second terminal and the first terminal switch the input device sharing mode to the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal, and the multi-screen collaboration window displays content associated with the first user interface.

[0308] In the embodiment of the present application, the first input device may be an input device such as a mouse, a stylus pen, a touchpad, etc., and no specific limitation is made here.

[0309] In one implementation, the first user interface is a user interface of a first application, and the multi-screen collaborative window displays the user interface of the first application. The method further includes: the second terminal detects a second operation acting on the multi-screen collaborative window; in response to the second operation, the second terminal displays the user interface of the second application in the multi-screen collaborative window. Fig. 9CAccording to the relevant description, the first application may be the aforementioned instant messaging application, the first user interface may be the aforementioned chat page 16, the first interface snapshot may be the aforementioned interface snapshot 303, the multi-screen collaborative window may be the aforementioned application continuation window 17, and the first operation may be the preset operation 5, for example, the first operation includes long pressing the left mouse button to select the chat page 16 and then moving the mouse.

[0310] In one implementation, the first user interface is a system interface. Fig.9A According to the relevant description, the first user interface can be the aforementioned main interface 13, the first interface snapshot can be the aforementioned interface snapshot 301, the multi-screen collaborative window can be the aforementioned projection window 15, and the first operation can be the preset operation 5. For example, the first operation includes long pressing the left mouse button to select the main interface 13 and then moving the mouse.

[0311] In some embodiments, the distance between the cursor and the first boundary of the first interface snapshot is a first value, and the distance between the cursor and the second boundary of the first interface snapshot is a second value; the first boundary is parallel to the second boundary, and the first boundary and the first boundary are respectively one of the left boundary, the right boundary, the upper boundary and the lower boundary; the above method also includes: when the cursor drags the first interface snapshot to move on the screen of the first terminal, when the distance between the cursor and the first boundary of the screen of the first terminal is less than the first value and continues to decrease, the first area of ​​the first interface snapshot moves out of the screen of the first terminal and moves into the screen of the second terminal; while the second terminal displays the first area of ​​the first interface snapshot, the first terminal displays the area outside the first area of ​​the first interface snapshot; when the cursor drags the first interface snapshot to move on the screen of the second terminal, when the distance between the cursor and the second boundary of the screen of the second terminal increases, the first area moved into the screen of the second terminal increases; when the distance between the cursor and the second boundary of the screen of the second terminal is less than the second value, while the second terminal displays the first area of ​​the first interface snapshot, the first terminal displays the area outside the first area of ​​the first interface snapshot. Exemplary, refer to Fig.9A The first boundary may be the right boundary, and the second boundary may be the left boundary.

[0312] In one implementation, the distance between the cursor and the boundary of the first interface snapshot is a first value, and the method further includes: when the cursor drags the first interface snapshot to move on the screen of the first terminal, when the distance between the cursor and the boundary of the screen of the first terminal is less than the first value and continues to decrease, the first area of ​​the first interface snapshot moves out of the screen of the first terminal and moves into the screen of the second terminal.

[0313] In one implementation, after the cursor drags the first interface snapshot to the screen of the second terminal, the first terminal displays the first area in the first interface snapshot, and the second terminal displays the second area excluding the first area in the first interface snapshot; alternatively, after the cursor drags the first interface snapshot to the screen of the second terminal, the first terminal does not display the first interface snapshot, and the second terminal displays a partial area in the first interface snapshot.

[0314] In one implementation, when the first user interface belongs to an application that cannot be applied to the second terminal, the multi-screen collaboration mode is a projection mode; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays the first user interface; when the first user interface belongs to the first application that can be applied to the second terminal, the multi-screen collaboration mode is an application continuation mode; the first user interface displayed by the first terminal is the user interface of the first version of the first application; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays the user interface corresponding to the first user interface in the second version of the first application; the first version is an application version applicable to the device type of the first terminal, and the second version is an application version applicable to the device type of the second terminal. In the embodiment of the present application, the first version can be the aforementioned version 1 (for example, the mobile phone version), and the second version can be the aforementioned version 2 (for example, the computer version).

[0315] In one implementation, the multi-screen collaboration mode is a projection mode; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays the first user interface. Fig.9A As described above, the mobile phone 100 displays the main interface 11, and the projection window 15 displays the main interface 11.

[0316] In one implementation, the multi-screen collaboration mode is an application continuation mode; the first user interface displayed by the first terminal is the user interface of the first application of the first version; the multi-screen collaboration window displays content associated with the first user interface, including: the multi-screen collaboration window displays the user interface corresponding to the first user interface in the second version of the first application; the first version is an application version suitable for the device type of the first terminal, and the second version is an application version suitable for the device type of the second terminal. Exemplary, reference Fig.9A , the mobile phone 100 displays a chat page 16 of "Henry" of the mobile version of the instant messaging application, and the application connection window 17 displays a chat page 19 of "Henry" of the computer version of the instant messaging application.

[0317] In one implementation, the first user interface belongs to a first application that can be applied to continue, and when the second terminal does not install the first application, before switching the input device sharing mode to the multi-screen collaboration mode, the method includes: the second terminal displays a prompt message, the prompt message is used to prompt the user to install the first application on the second terminal; in response to a third operation, the second terminal installs the first application and determines that the multi-screen collaboration mode is the application continuation mode; in response to a fourth operation, the multi-screen collaboration mode is determined to be the projection mode. Exemplary, reference Fig. 4I The prompt information may be the aforementioned prompt information 113; the third operation is used to install the first application (eg, click the operation of installing the control 114); the fourth operation may be used to cancel the installation of the first application (eg, click the operation of canceling the control 115).

[0318] In one implementation, the method also includes: when the first terminal and the second terminal are in a multi-screen collaborative mode, in response to a fifth operation of the first input device, the second terminal displays the cursor of the first input device dragging the multi-screen collaborative window to move; during the process of dragging the multi-screen collaborative window with the cursor, after the cursor moves to the boundary of the screen of the second terminal, the second terminal and the first terminal switch the multi-screen collaborative mode to the input device sharing mode, the second terminal stops displaying the multi-screen collaborative window, and shares the above-mentioned at least one input device with the first terminal.

[0319] In one implementation, the multi-screen collaboration window displays a second user interface; when the multi-screen collaboration mode is switched to the input device sharing mode, the method further includes: the first terminal displays content associated with the second user interface.

[0320] In one implementation, the multi-screen collaboration mode is a projection mode, and the first terminal displays content associated with the second user interface, including: the first terminal displays the second user interface. Fig. 9B According to the relevant description, the second user interface can be the aforementioned chat page 16, the multi-screen collaborative window can be the aforementioned projection window 15, and the fifth operation includes long pressing the left mouse button to select the projection window 15 and then moving the mouse.

[0321] In one implementation, the multi-screen collaboration mode is an application continuation mode, the second user interface displayed in the multi-screen collaboration window is the user interface of the second version of the first application, and the first terminal displays content associated with the second user interface, including: the first terminal displays the user interface corresponding to the second user interface in the first version of the first application; the first version is an application version suitable for the device type of the first terminal; the second version is an application version suitable for the device type of the second terminal. Exemplary, refer to Fig.9DAccording to the relevant description, the first application may be the aforementioned instant messaging application, the second user interface may be the chat page 19 of the aforementioned contact “Henry”, and the multi-screen collaborative window may be the aforementioned application connection window 17.

[0322] In one implementation, the method further includes: when the first terminal and the second terminal are not in the input device sharing mode and the multi-screen collaboration mode, the first terminal or the second terminal receives and responds to the sixth operation, and when the device state of the first terminal meets the first start condition, the first terminal and the second terminal are triggered to start the input device sharing mode, and the second terminal shares the at least one input device with the first terminal. Exemplary, reference Figure 4C and Figure 4D , according to the device status of the mobile phone 100, you can choose to start the input device sharing mode. When the device status of the first terminal meets the second start condition, the first terminal and the second terminal are triggered to start the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal. For example, refer to FIG. 4E to FIG. 4H According to the relevant description, according to the device status of the mobile phone 100, you can choose to start the multi-screen collaboration mode.

[0323] In the embodiment of the present application, the sixth operation may be the aforementioned preset operation 1, the first start condition may be the aforementioned start condition 1, and the second start condition may be the aforementioned start condition 2.

[0324] In one implementation, the method further includes: when the first terminal and the second terminal are in the multi-screen collaborative mode, when it is detected that the device state of the first terminal satisfies the first starting condition, triggering the first terminal and the second terminal to switch the multi-screen collaborative mode to the input device sharing mode. When the first terminal and the second terminal are in the input device sharing mode, when it is detected that the device state of the first terminal satisfies the second starting condition, triggering the first terminal and the second terminal to switch the input device sharing mode to the multi-screen collaborative mode. Exemplary, reference Figure 7 and Figure 8 According to the relevant description, the collaborative mode can be automatically switched according to the device status of the mobile phone 100.

[0325] In one implementation, the first terminal and the second terminal switch the input device sharing mode to the multi-screen collaboration mode, including: when the second terminal cannot apply the continuation of the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the screen projection mode, and the second terminal displays the screen projection content from the first terminal in the multi-screen collaboration window; when the second terminal can apply the continuation of the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the application continuation mode, and the second terminal performs the application continuation of the application running in the foreground of the first terminal in the multi-screen collaboration window. Exemplary, refer to Figure 7 and Figure 8 , when automatically switching to the multi-screen collaboration mode, the multi-screen collaboration mode can be automatically determined as the projection mode or the application continuation mode according to the page running in the foreground.

[0326] In one implementation, the first start condition includes one or more of the following: not in the accidental touch prevention scene, the accidental touch prevention function is not turned on, the screen faces the user, the posture tends to be upright, and the distance from the second terminal is less than a third value; the second start condition includes: the device state of the first terminal does not meet the first start condition. The third value can be the aforementioned distance 2.

[0327] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0328] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0329] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0330] In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-device collaboration method, It is characterized in that Applied to a multi-device collaborative system, the multi-device collaborative system includes a first terminal and a second terminal, the second terminal is connected to a first input device, and the method includes: The first terminal establishes a communication connection with the second terminal; The first terminal displays a first user interface; The second terminal shares at least one input device with the first terminal, the first terminal and the second terminal are in an input device sharing mode, and the at least one input device includes the first input device; In response to a first operation of the first input device, the first terminal displays a first interface snapshot of the first user interface being dragged by a cursor of the first input device to move; After the cursor drags the first interface snapshot to the display screen of the second terminal, the second terminal and the first terminal switch the input device sharing mode to the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal, and the multi-screen collaboration window displays content associated with the first user interface.

2. The method according to claim 1, It is characterized in that The first user interface is a user interface of a first application, and the multi-screen collaboration window displays the user interface of the first application. The method further includes: The second terminal detects a second operation acting on the multi-screen collaborative window; In response to the second operation, the second terminal displays a user interface of a second application in the multi-screen collaborative window.

3. The method according to claim 1, It is characterized in that The first user interface is a system interface.

4. The method according to claim 1, It is characterized in that The distance between the cursor and the boundary of the first interface snapshot is a first value, and the method further includes: When the cursor drags the first interface snapshot on the screen of the first terminal, when the distance between the cursor and the boundary of the screen of the first terminal is less than the first value and continues to decrease, the first area of ​​the first interface snapshot moves out of the screen of the first terminal and moves into the screen of the second terminal.

5. The method according to claim 1, It is characterized in that After the cursor drags the first interface snapshot to the screen of the second terminal, the first terminal displays the first area in the first interface snapshot, and the second terminal displays the second area in the first interface snapshot except the first area, or, After the cursor drags the first interface snapshot to the screen of the second terminal, the first terminal does not display the first interface snapshot, and the second terminal displays a partial area of ​​the first interface snapshot.

6. The method according to claim 1, It is characterized in that When the first user interface belongs to an application that the second terminal cannot apply to continue, the multi-screen collaboration mode is a projection mode; and the multi-screen collaboration window displays content associated with the first user interface, including: The multi-screen collaborative window displays the first user interface; When the first user interface belongs to the first application that can be applied and continued by the second terminal, the multi-screen collaboration mode is an application continuation mode; the first user interface displayed by the first terminal is a user interface of the first version of the first application; and the multi-screen collaboration window displays content associated with the first user interface, including: The multi-screen collaborative window displays a user interface corresponding to the first user interface in the second version of the first application; the first version is an application version suitable for the device type of the first terminal, and the second version is an application version suitable for the device type of the second terminal.

7. The method according to claim 1, It is characterized in that The first user interface belongs to a first application that can be applied continuously, and when the second terminal does not install the first application, before switching the input device sharing mode to the multi-screen collaboration mode, the method includes: The second terminal displays prompt information, where the prompt information is used to prompt the user to install the first application on the second terminal; In response to the third operation, the second terminal installs the first application and determines that the multi-screen collaboration mode is an application continuation mode; In response to the fourth operation, it is determined that the multi-screen collaborative mode is a screen projection mode.

8. The method according to claim 1, It is characterized in that The method further comprises: When the first terminal and the second terminal are in the multi-screen collaborative mode, in response to a fifth operation of the first input device, the second terminal displays the cursor of the first input device dragging the multi-screen collaborative window to move; During the process of dragging the multi-screen collaborative window with the cursor, after the cursor moves to the boundary of the screen of the second terminal, the second terminal and the first terminal switch the multi-screen collaborative mode to the input device sharing mode, the second terminal stops displaying the multi-screen collaborative window, and shares the at least one input device with the first terminal.

9. The method according to claim 8, It is characterized in that The multi-screen collaborative window displays a second user interface; When the multi-screen collaboration mode is switched to the input device sharing mode, the method further includes: The first terminal displays content associated with the second user interface.

10. The method according to claim 9, It is characterized in that The multi-screen collaboration mode is a screen projection mode, and the first terminal displays content associated with the second user interface, including: The first terminal displays the second user interface.

11. The method according to claim 9, It is characterized in that The multi-screen collaboration mode is an application continuation mode, the second user interface displayed in the multi-screen collaboration window is a user interface of a second version of the first application, and the first terminal displays content associated with the second user interface, including: The first terminal displays a user interface in the first version of the first application corresponding to the second user interface; the first version is an application version suitable for the device type of the first terminal; the second version is an application version suitable for the device type of the second terminal.

12. The method according to any one of claims 1 to 11, It is characterized in that The method further comprises: When the first terminal and the second terminal are not in the input device sharing mode and the multi-screen collaboration mode, the first terminal or the second terminal receives and responds to a sixth operation, and when the device state of the first terminal meets a first starting condition, triggers the first terminal and the second terminal to start the input device sharing mode, and the second terminal shares the at least one input device with the first terminal; When the device state of the first terminal meets the second starting condition, the first terminal and the second terminal are triggered to start the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal.

13. The method according to any one of claims 1 to 12, It is characterized in that The method further comprises: When the first terminal and the second terminal are in the multi-screen collaborative mode, when it is detected that the device state of the first terminal meets the first starting condition, triggering the first terminal and the second terminal to switch the multi-screen collaborative mode to the input device sharing mode; When the first terminal and the second terminal are in the input device sharing mode, when it is detected that the device state of the first terminal meets the second starting condition, the first terminal and the second terminal are triggered to switch the input device sharing mode to the multi-screen collaboration mode.

14. The method according to claim 12 or 13, It is characterized in that The first terminal and the second terminal switch the input device sharing mode to the multi-screen collaboration mode, including: When the second terminal cannot connect to the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the screen projection mode, and the second terminal displays the screen projection content from the first terminal in the multi-screen collaborative window; When the second terminal can perform application continuation on the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the application continuation mode, and the second terminal performs application continuation on the application running in the foreground of the first terminal in the multi-screen collaborative window.

15. The method according to any one of claims 12 to 14, It is characterized in that The first start condition includes one or more of the following: not in the accidental touch prevention scene, the accidental touch prevention function is not turned on, the screen faces the user, the posture tends to be upright, and the distance from the second terminal is less than a third value; The second start condition includes: a device state of the first terminal does not satisfy the first start condition.

16. A multi-device collaboration method, It is characterized in that The method comprises: The first terminal establishes a communication connection with the second terminal, and the second terminal is connected to a first input device; The first terminal displays a first user interface; The first terminal and the second terminal are in an input device sharing mode, the first terminal shares at least one input device of the second terminal, and the at least one input device includes the first input device; In response to a first operation of the first input device, the first terminal displays a first interface snapshot of the first user interface being dragged by a cursor of the first input device to move; After the cursor drags the first interface snapshot out of the screen of the first terminal, the first terminal switches the input device sharing mode with the second terminal to the multi-screen collaboration mode, and sends collaboration data of the multi-screen collaboration window to the second terminal, the collaboration data is used to determine the display content of the multi-screen collaboration window, and the multi-screen collaboration window is used to be displayed on the screen of the second terminal. The multi-screen collaboration window is used to display content associated with the first user interface.

17. The method according to claim 16, It is characterized in that The first user interface is a user interface of a first application, the multi-screen collaboration window is used to display content in the first application that is associated with the first user interface, and the multi-screen collaboration window is also used to display a user interface of a second application.

18. The method according to claim 16, It is characterized in that The first user interface is a system interface.

19. The method according to claim 16, It is characterized in that When the first user interface belongs to an application that the second terminal cannot apply to continue, the multi-screen collaboration mode is a projection mode; and the content associated with the first user interface is the first user interface; When the first user interface belongs to the first application that can be applied and continued by the second terminal, the multi-screen collaboration mode is an application continuation mode; the first user interface displayed by the first terminal is a user interface of the first version of the first application; The content associated with the first user interface includes a user interface corresponding to the first user interface in the second version of the first application; The first version is an application version applicable to a device type of the first terminal, and the second version is an application version applicable to a device type of the second terminal.

20. The method according to claim 16, It is characterized in that The method further comprises: When the first terminal and the second terminal are in the multi-screen collaborative mode, in response to the first information sent by the second terminal, the first terminal switches the multi-screen collaborative mode with the second terminal to the input device sharing mode; The first information is sent after the cursor drags the multi-screen collaborative window to move on the screen of the second terminal and the cursor moves to the boundary of the screen of the second terminal.

21. The method according to claim 20, It is characterized in that The multi-screen collaborative window displays a second user interface; when the multi-screen collaborative mode with the second terminal is switched to the input device sharing mode, the method further includes: The first terminal displays content associated with the second user interface.

22. The method according to claim 21, It is characterized in that The multi-screen collaboration mode is a screen projection mode, and the first terminal displays content associated with the second user interface, including: The first terminal displays the second user interface.

23. The method according to claim 21, It is characterized in that The multi-screen collaboration mode is an application continuation mode, the second user interface displayed in the multi-screen collaboration window is a user interface of a second version of the first application, and the first terminal displays content associated with the second user interface, including: The first terminal displays a user interface in the first version of the first application corresponding to the second user interface; the first version is an application version suitable for the device type of the first terminal; the second version is an application version suitable for the device type of the second terminal.

24. A multi-device collaboration method, It is characterized in that The method comprises: The second terminal establishes a communication connection with the first terminal, and the second terminal is connected to a first input device; The second terminal shares at least one input device with the first terminal, the first terminal and the second terminal are in an input device sharing mode, and the at least one input device includes the first input device; After the cursor of the first input device drags the first interface snapshot of the first user interface displayed by the first terminal to the display screen of the second terminal, the second terminal switches the input device sharing mode with the first terminal to the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal, and the multi-screen collaboration window displays content associated with the first user interface.

25. The method according to claim 24, It is characterized in that The first user interface is a user interface of a first application, and the multi-screen collaboration window displays the user interface of the first application. The method further includes: The second terminal detects a second operation acting on the multi-screen collaborative window; In response to the second operation, the second terminal displays a user interface of a second application in the multi-screen collaborative window.

26. The method according to claim 24, It is characterized in that The first user interface is a system interface.

27. The method according to claim 24, It is characterized in that When the first user interface belongs to an application that the second terminal cannot apply to continue, the multi-screen collaboration mode is a projection mode; and the multi-screen collaboration window displays content associated with the first user interface, including: The multi-screen collaborative window displays the first user interface; When the first user interface belongs to the first application that can be applied and continued by the second terminal, the multi-screen collaboration mode is an application continuation mode; the first user interface displayed by the first terminal is a user interface of the first version of the first application; and the multi-screen collaboration window displays content associated with the first user interface, including: The multi-screen collaborative window displays a user interface corresponding to the first user interface in the second version of the first application; the first version is an application version suitable for the device type of the first terminal, and the second version is an application version suitable for the device type of the second terminal.

28. The method according to claim 24, It is characterized in that The first user interface belongs to a first application that can be applied continuously, and when the second terminal does not install the first application, before switching the input device sharing mode to the multi-screen collaboration mode, the method includes: The second terminal displays prompt information, where the prompt information is used to prompt the user to install the first application on the second terminal; In response to the third operation, the second terminal installs the first application and determines that the multi-screen collaboration mode is an application continuation mode; In response to the fourth operation, it is determined that the multi-screen collaborative mode is a screen projection mode.

29. The method according to claim 24, It is characterized in that The method further comprises: When the first terminal and the second terminal are in the multi-screen collaborative mode, in response to a fifth operation of the first input device, the second terminal displays the cursor of the first input device dragging the multi-screen collaborative window to move; During the process of dragging the multi-screen collaborative window with the cursor, after the cursor moves to the boundary of the screen of the second terminal, the second terminal and the first terminal switch the multi-screen collaborative mode to the input device sharing mode, the second terminal stops displaying the multi-screen collaborative window, and shares the at least one input device with the first terminal.

30. A multi-device collaboration method, It is characterized in that Applied to a multi-device collaborative system, the multi-device collaborative system includes a first terminal and a second terminal, and the method includes: When the first terminal and the second terminal are not in the input device sharing mode and the multi-screen collaboration mode, the first terminal or the second terminal receives and responds to a sixth operation, and when the device state of the first terminal meets a first starting condition, triggers the first terminal and the second terminal to start the input device sharing mode, and the second terminal shares the at least one input device with the first terminal; When the device state of the first terminal meets the second starting condition, the first terminal and the second terminal are triggered to start the multi-screen collaboration mode, and the second terminal displays the multi-screen collaboration window corresponding to the first terminal.

31. The method according to claim 30, It is characterized in that The method further comprises: When the first terminal and the second terminal are in the multi-screen collaborative mode, when it is detected that the device state of the first terminal meets the first starting condition, triggering the first terminal and the second terminal to switch the multi-screen collaborative mode to the input device sharing mode; When the first terminal and the second terminal are in the input device sharing mode, when it is detected that the device state of the first terminal meets the second starting condition, the first terminal and the second terminal are triggered to switch the input device sharing mode to the multi-screen collaboration mode.

32. The method according to claim 30 or 31, It is characterized in that The first terminal and the second terminal switch the input device sharing mode to the multi-screen collaboration mode, including: When the second terminal cannot connect to the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the screen projection mode, and the second terminal displays the screen projection content from the first terminal in the multi-screen collaborative window; When the second terminal can perform application continuation on the application running in the foreground of the first terminal, the first terminal and the second terminal switch the input device sharing mode to the application continuation mode, and the second terminal performs application continuation on the application running in the foreground of the first terminal in the multi-screen collaborative window.

33. The method according to any one of claims 30 to 31, It is characterized in that The first start condition includes one or more of the following: not in the accidental touch prevention scene, the accidental touch prevention function is not turned on, the screen faces the user, the posture tends to be upright, and the distance from the second terminal is less than a third value; The second start condition includes: a device state of the first terminal does not satisfy the first start condition.

34. A terminal device, It is characterized in that include: A processor and a memory, wherein the memory is coupled to the processor, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor reads the computer instructions from the memory, the terminal device executes the multi-device collaboration method as described in any one of claims 16 to 23 or 24 to 29.

35. A computer readable storage medium, It is characterized in that It includes computer instructions, which, when executed on a terminal, enable the terminal device to execute a multi-device collaboration method as described in any one of claims 16 to 23 or 24 to 29.

Images