Collaborative display method, terminal equipment and computer readable storage medium

Abstract

The invention is suitable for the technical field of terminals, and particularly relates to a collaborative display method, terminal equipment and a computer readable storage medium. According to the method, a first display area of a main screen of first terminal equipment in second terminal equipment and a second display area of a virtual screen in the second terminal equipment can be determined, so that the second terminal equipment can display the main screen in the first display area and display the virtual screen in the second display area. When the display state of the first terminal equipment changes, only the content display mode of the main screen, the first display area and the second display area can be dynamically adjusted, and the content display mode of the virtual screen and the relative position of the virtual screen in the second display area do not need to be adjusted; the content of the application program displayed on the virtual screen does not need to be re-arranged and refreshed, so that the display time delay of the content displayed in the second display area is reduced, the user experience is improved, and the method has high usability and practicability.

Description

[0001] This application is a divisional application. The application number of the original application is 202011596431.0, and the original application date is December 28, 2020. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present application belongs to the field of terminal technology, and in particular, relates to a collaborative display method, a terminal device, and a computer-readable storage medium. Background Art

[0003] Collaborative display refers to the process of completing a certain display service in a coordinated and consistent manner through two or more terminal devices. In the existing collaborative display, multiple display windows can be set on the second terminal device (such as a tablet computer) to display multiple applications of the first terminal device (such as a mobile phone). However, when the display state of the first terminal device changes (such as from a vertical screen display state to a horizontal screen display state), the display window of the second terminal device will also change accordingly (such as from a vertical window to a horizontal window), so that the content displayed in each display window needs to be rearranged and refreshed, causing a delay in the display of content in the second terminal device, affecting the user experience. Summary of the invention

[0004] The embodiments of the present application provide a collaborative display method, a terminal device, and a computer-readable storage medium. In collaborative display, when the display state of a first terminal device changes, the re-layout and refresh of the content can be reduced, thereby reducing the delay in content display in a second terminal device and improving the user experience.

[0005] In a first aspect, an embodiment of the present application provides a collaborative display method, which is applied to a first terminal device. The method may include:

[0006] Determine, according to the first display state of the first terminal device, a first display area of ​​the first main screen of the first terminal device in the second terminal device and a second display area of ​​the virtual screen of the first terminal device in the second terminal device;

[0007] Casting the first main screen to the second terminal device, and sending first position information of the first display area to the second terminal device, where the first position information is used by the second terminal device to display the first main screen in the first display area;

[0008] Projecting the virtual screen to the second terminal device, and sending second position information of the second display area to the second terminal device, where the second position information is used by the second terminal device to display the virtual screen in the second display area;

[0009] When detecting that the first terminal device is adjusted from the first display state to the second display state, adjusting the content display mode of the first main screen according to the second display state to obtain the second main screen, and reorganizing the first display area and the second display area according to the second display state;

[0010] The second main screen and the virtual screen are projected to the second terminal device, and the third position information of the reorganized first display area and the fourth position information of the reorganized second display area are sent to the second terminal device, the third position information is used by the second terminal device to display the second main screen in the reorganized first display area, and the fourth position information is used by the second terminal device to display the virtual screen in the reorganized second display area.

[0011] Through the above-mentioned collaborative display method, the first terminal device can determine the first display area of ​​the main screen of the first terminal device in the second terminal device and the second display area of ​​the virtual screen in the second terminal device according to the display state of the first terminal device, so that the second terminal device can display the main screen in the first display area and the virtual screen in the second display area. When the display state of the first terminal device changes, the first terminal device can only dynamically adjust the content display mode of the main screen and the first display area and the second display area, without adjusting the content display mode of the virtual screen and the relative position of the virtual screen in the second display area, so that there is no need to re-layout and refresh the content of the application displayed on the virtual screen, so as to reduce the display delay of the content displayed in the second display area, improve the user experience, and have strong ease of use and practicality.

[0012] Optionally, before projecting the virtual screen to the second terminal device, the method may include:

[0013] When obtaining the application startup instruction sent by the second terminal device, the virtual screen is created according to the first main screen, and the application interface of the application is displayed on the virtual screen, and the virtual screen is not displayed in the first terminal device.

[0014] In the collaborative display method provided by this implementation, when a user clicks on an application in the first main screen of the second terminal device, the second terminal device can send an instruction to open the application to the first terminal device. At this time, the first terminal device can create a virtual screen, display the application interface of the application on the virtual screen, and project the virtual screen to the second display area of ​​the second terminal device, so that multiple windows of the first terminal device can be opened simultaneously on the second terminal device, which facilitates user browsing and improves user experience.

[0015] In a possible implementation manner of the first aspect, determining, according to the first display state of the first terminal device, a first display area of ​​the first main screen of the first terminal device in the second terminal device and a second display area of ​​the virtual screen of the first terminal device in the second terminal device may include:

[0016] Acquire interface information of the second terminal device;

[0017] The first display area and the second display area are determined according to a first display state of the first terminal device and interface information of the second terminal device.

[0018] In the collaborative display method provided by this implementation, the first terminal device can accurately determine the first display area and the second display area based on the interface information of the second terminal device and the first display state of the first terminal device to ensure that the second terminal device can display more windows of the first terminal device, improve the utilization rate of the display interface in the second terminal device, and enhance the user's browsing experience.

[0019] Exemplarily, the interface information includes an interface height and an interface width, and determining the first display area and the second display area according to the first display state of the first terminal device and the interface information of the second terminal device may include:

[0020] When the first display state is a vertical screen display state, determining that a first area height of the first display area is the same as a second area height of the second display area, and that the first area height and the second area height are the same as a height of the interface;

[0021] It is determined that a first area width of the first display area is the same as a screen width of the first main screen.

[0022] Exemplarily, the interface information includes an interface height and an interface width, and determining the first display area and the second display area according to the first display state of the first terminal device and the interface information of the second terminal device may include:

[0023] When the first display state is a horizontal screen display state, determining that a first area width of the first display area is the same as a second area width of the second display area, and the first area width and the second area width are the same as the interface width;

[0024] It is determined that a first area height of the first display area is the same as a screen height of the first main screen.

[0025] In another possible implementation manner of the first aspect, the first terminal device is a terminal device with a foldable display screen, and the interface information includes an interface height and an interface width;

[0026] The determining the first display area and the second display area according to the first display state of the first terminal device and the interface information of the second terminal device may include:

[0027] When the first display state is an expanded screen display state, determining a third display state of the second terminal device according to the interface height and the interface width;

[0028] The first display area and the second display area are determined according to the third display state and the interface information.

[0029] In the collaborative display method provided by this implementation, when the first terminal device with a foldable display screen is in an expanded screen display state, the main screen of the first terminal device will become larger accordingly, and the area of ​​the first display area also needs to become larger accordingly. In order to make full use of the display interface of the second terminal device and improve the utilization rate of the display interface in the second terminal device, the first terminal device can also determine the first display area and the second display area in combination with the display status and interface information of the second terminal device.

[0030] Optionally, the determining the first display area and the second display area according to the third display state and the interface information may include:

[0031] When the third display state is a horizontal screen display state, determining that a first area height of the first display area is the same as a second area height of the second display area, and that the first area height, the second area height and the interface height are the same;

[0032] It is determined that a first area width of the first display area is the same as a screen width of the first main screen, or is the same as a screen height of the first main screen.

[0033] Optionally, the determining the first display area and the second display area according to the third display state and the interface information may include:

[0034] When the third display state is a vertical screen display state, determining that a first area width of the first display area is the same as a second area width of the second display area, and the first area width and the second area width are the same as the interface width;

[0035] It is determined that a first area height of the first display area is the same as a screen height of the first main screen, or is the same as a screen width of the first main screen.

[0036] In a second aspect, an embodiment of the present application provides a collaborative display method, which is applied to a second terminal device. The method may include:

[0037] Acquire a first display state of the first terminal device, and determine a first display area of ​​a first main screen of the first terminal device in the second terminal device and a second display area of ​​a virtual screen of the first terminal device in the second terminal device according to the first display state;

[0038] Obtaining a first main screen projected by the first terminal device, and displaying the first main screen in the first display area;

[0039] Obtaining a virtual screen projected by the first terminal device, and displaying the virtual screen in the second display area;

[0040] When it is determined that the first terminal device is adjusted from the first display state to the second display state, reorganizing the first display area and the second display area according to the second display state;

[0041] Obtain the second main screen and the virtual screen projected by the first terminal device, and display the second main screen in the reorganized first display area, and display the virtual screen in the reorganized second display area. The content display method of the second main screen corresponds to the second display state.

[0042] Through the above-mentioned collaborative display method, the second terminal device can determine the first display area of ​​the main screen of the first terminal device in the second terminal device and the second display area of ​​the virtual screen in the second terminal device according to the display state of the first terminal device, and can display the main screen in the first display area and the virtual screen in the second display area. When the display state of the first terminal device changes, the second terminal device can only dynamically adjust the first display area and the second display area without adjusting the content display method of the virtual screen and the relative position of the virtual screen in the second display area, so that there is no need to re-layout and refresh the content of the application displayed on the virtual screen, so as to reduce the display delay of the content displayed in the second display area, improve the user experience, and have strong ease of use and practicality.

[0043] Optionally, before obtaining the virtual screen projected by the first terminal device, the method may include:

[0044] In response to a start-up operation on an application in the first main screen, a start-up instruction for starting the application is sent to the first terminal device, where the start-up instruction is used by the first terminal device to start the application and display the application interface of the application on the virtual screen.

[0045] In a possible implementation manner of the second aspect, determining, according to the first display state, a first display area of ​​a first main screen of the first terminal device in the second terminal device and a second display area of ​​a virtual screen of the first terminal device in the second terminal device may include:

[0046] Acquire interface information of the second terminal device;

[0047] The first display area and the second display area are determined according to the first display state and the interface information of the second terminal device.

[0048] Exemplarily, the interface information includes an interface height and an interface width, and determining the first display area and the second display area according to the first display state and the interface information of the second terminal device may include:

[0049] When the first display state is a vertical screen display state, determining that a first area height of the first display area is the same as a second area height of the second display area, and that the first area height and the second area height are the same as a height of the interface;

[0050] It is determined that a first area width of the first display area is the same as a screen width of the first main screen.

[0051] Exemplarily, the interface information includes an interface height and an interface width, and determining the first display area and the second display area according to the first display state and the interface information of the second terminal device may include:

[0052] When the first display state is a horizontal screen display state, determining that a first area width of the first display area is the same as a second area width of the second display area, and the first area width and the second area width are the same as the interface width;

[0053] It is determined that a first area height of the first display area is the same as a screen height of the first main screen.

[0054] In another possible implementation manner of the second aspect, the first terminal device is a terminal device with a foldable display screen, and the interface information includes an interface height and an interface width;

[0055] The determining the first display area and the second display area according to the first display state and the interface information of the second terminal device may include:

[0056] When the first display state is an expanded screen display state, determining a third display state of the second terminal device according to the interface height and the interface width;

[0057] The first display area and the second display area are determined according to the third display state and the interface information.

[0058] Optionally, the determining the first display area and the second display area according to the third display state and the interface information may include:

[0059] When the third display state is a horizontal screen display state, determining that a first area height of the first display area is the same as a second area height of the second display area, and that the first area height, the second area height and the interface height are the same;

[0060] It is determined that a first area width of the first display area is the same as a screen width of the first main screen, or is the same as a screen height of the first main screen.

[0061] Optionally, the determining the first display area and the second display area according to the third display state and the interface information may include:

[0062] When the third display state is a vertical screen display state, determining that a first area width of the first display area is the same as a second area width of the second display area, and the first area width and the second area width are the same as the interface width;

[0063] It is determined that a first area height of the first display area is the same as a screen height of the first main screen, or is the same as a screen width of the first main screen.

[0064] In a third aspect, an embodiment of the present application provides a collaborative display device, applied to a first terminal device, and the device may include:

[0065] A display area determination module, configured to determine a first display area of ​​a first main screen of the first terminal device in a second terminal device and a second display area of ​​a virtual screen of the first terminal device in the second terminal device according to a first display state of the first terminal device;

[0066] A main screen projection module, used for projecting the first main screen to the second terminal device, and sending first position information of the first display area to the second terminal device, wherein the first position information is used by the second terminal device to display the first main screen in the first display area;

[0067] a virtual screen projection module, configured to project the virtual screen to the second terminal device, and send second position information of the second display area to the second terminal device, wherein the second position information is used by the second terminal device to display the virtual screen in the second display area;

[0068] A display area reorganization module is used to adjust the content display mode of the first main screen according to the second display state to obtain the second main screen when detecting that the first terminal device is adjusted from the first display state to the second display state, and reorganize the first display area and the second display area according to the second display state;

[0069] A collaborative display module is used to project the second main screen and the virtual screen to the second terminal device, and send the third position information of the reorganized first display area and the fourth position information of the reorganized second display area to the second terminal device, the third position information is used by the second terminal device to display the second main screen in the reorganized first display area, and the fourth position information is used by the second terminal device to display the virtual screen in the reorganized second display area.

[0070] Optionally, the device may further include:

[0071] A virtual screen creation module is used to create the virtual screen according to the first main screen when obtaining the startup instruction of the application sent by the second terminal device, and display the application interface of the application on the virtual screen. The virtual screen is not displayed in the first terminal device.

[0072] In a possible implementation manner of the third aspect, the display area determining module may include:

[0073] An interface information acquisition unit, used to acquire interface information of the second terminal device;

[0074] The display area determining unit is configured to determine the first display area and the second display area according to the first display state of the first terminal device and the interface information of the second terminal device.

[0075] Exemplarily, the interface information includes interface height and interface width;

[0076] The display area determination unit is used to determine that, when the first display state is a vertical screen display state, the first area height of the first display area is the same as the second area height of the second display area, and the first area height and the second area height are the same as the interface height; and determine that the first area width of the first display area is the same as the screen width of the first main screen.

[0077] Exemplarily, the interface information includes interface height and interface width;

[0078] The display area determination unit is also used to determine that, when the first display state is a horizontal screen display state, a first area width of the first display area is the same as a second area width of the second display area, and the first area width and the second area width are the same as the interface width; and determine that a first area height of the first display area is the same as a screen height of the first main screen.

[0079] In another possible implementation manner of the third aspect, the first terminal device is a terminal device with a foldable display screen, and the interface information includes an interface height and an interface width;

[0080] The display area determination unit may include:

[0081] A display state determination subunit, configured to determine a third display state of the second terminal device according to the interface height and the interface width when the first display state is an expanded screen display state;

[0082] The display area determination subunit is used to determine the first display area and the second display area according to the third display state and the interface information.

[0083] Optionally, the display area determination subunit is used to determine that the first area height of the first display area is the same as the second area height of the second display area, and the first area height, the second area height and the interface height are the same; determine that the first area width of the first display area is the same as the screen width of the first terminal device, or the same as the screen height of the first terminal device when the third display state is a horizontal screen display state.

[0084] Optionally, the display area determination subunit is also used to determine that, when the third display state is a vertical screen display state, a first area width of the first display area is the same as a second area width of the second display area, and the first area width and the second area width are the same as the interface width; and to determine that a first area height of the first display area is the same as a screen height of the first terminal device, or is the same as a screen width of the first terminal device.

[0085] In a fourth aspect, an embodiment of the present application provides a collaborative display device, which is applied to a second terminal device. The device may include:

[0086] A display area determination module, configured to obtain a first display state of a first terminal device, and determine a first display area of ​​a first main screen of the first terminal device in the second terminal device and a second display area of ​​a virtual screen of the first terminal device in the second terminal device according to the first display state;

[0087] A main screen acquisition module, used to acquire a first main screen projected by the first terminal device, and display the first main screen in the first display area;

[0088] A virtual screen acquisition module, used to acquire the virtual screen projected by the first terminal device, and display the virtual screen in the second display area;

[0089] A display area reorganization module, configured to reorganize the first display area and the second display area according to the second display state when it is determined that the first terminal device is adjusted from the first display state to the second display state;

[0090] A collaborative display module is used to obtain the second main screen and the virtual screen projected by the first terminal device, and display the second main screen in the reorganized first display area, and display the virtual screen in the reorganized second display area. The content display method of the second main screen corresponds to the second display state.

[0091] Optionally, the device may further include:

[0092] A startup instruction sending module is used to send a startup instruction for starting the application in the first main screen to the first terminal device in response to a startup operation on the application in the first main screen, wherein the startup instruction is used for the first terminal device to start the application and display the application interface of the application on the virtual screen.

[0093] In the fifth aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the terminal device implements any one of the methods described in the first aspect above.

[0094] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the computer implements any one of the methods described in the first aspect above.

[0095] In a seventh aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any of the methods described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 is a schematic diagram of the structure of a terminal device to which the collaborative display method provided in an embodiment of the present application is applicable;

[0097] Figure 2 is a schematic diagram of a software architecture to which the collaborative display method provided in an embodiment of the present application is applicable;

[0098] Figure 3 is a schematic diagram of a system provided by an embodiment of the present application;

[0099] Figure 4 and Figure 5 is a schematic diagram of establishing a connection between a first terminal device and a second terminal device;

[0100] Figure 6 is an example of determining the first display area and the second display area Figure 1 ;

[0101] Figure 7 is an example of determining the first display area and the second display area Figure 2 ;

[0102] Figure 8 is an example of determining the first display area and the second display area Figure 3 ;

[0103] Fig. 9 is an example of determining the first display area and the second display area Figure 4 ;

[0104] Fig.10 is an example of reorganizing the first display area and the second display area Figure 1 ;

[0105] Fig.11 is an example of reorganizing the first display area and the second display area Figure 2 ;

[0106] Fig.12 is an example of reorganizing the first display area and the second display area Figure 3 ;

[0107] Fig.13 is an example of reorganizing the first display area and the second display area Figure 4 ;

[0108] Fig.14 is an example of reorganizing the first display area and the second display area Figure 5 ;

[0109] Figures 15 to 17 This is an example of an application scenario provided by an embodiment of the present application. Figure 1 ;

[0110] Figures 18 to 21 This is an example of an application scenario provided by an embodiment of the present application. Figure 2 ;

[0111] Fig. 22 It is a flowchart of a collaborative display method provided in one embodiment of the present application. DETAILED DESCRIPTION

[0112] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0113] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0114] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0115] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0116] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0117] In addition, the “plurality” mentioned in the embodiments of the present application should be interpreted as two or more.

[0118] The steps involved in the collaborative display method provided in the embodiments of the present application are only examples. Not all steps are mandatory steps, or not all information or messages are mandatory. They can be increased or decreased as needed during use. The same step or steps or messages with the same function in different embodiments of the present application can be referenced and learned from each other.

[0119] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art will appreciate that, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0120] As introduced in the background technology section, in the existing collaborative display, when multiple display windows are set on a second terminal device (such as a tablet computer) to display multiple applications of a first terminal device (such as a mobile phone), if the display state of the first terminal device changes (such as from a vertical screen display state to a horizontal screen display state), the display window of the second terminal device will also change accordingly (such as from a vertical window to a horizontal window), thereby requiring the content displayed in each display window to be rearranged and refreshed, causing a delay in the content display in the second terminal device, affecting the user experience.

[0121] To solve the above problems, the embodiment of the present application provides a collaborative display method. During collaborative display, the first terminal device can create a virtual screen independent of the main screen, and can send the display information of the main screen and the virtual screen to the second terminal device respectively. The second terminal device can display the main screen in the first display area and the virtual screen in the second display area. When the display state of the first terminal device changes, only the content display mode of the main screen and the first display area and the second display area can be dynamically adjusted without adjusting the content display mode of the virtual screen and the relative position of the virtual screen in the second display area, so that there is no need to rearrange and refresh the content of the application displayed on the virtual screen, so as to reduce the display delay of the content displayed in the second display area, improve the user experience, and have strong ease of use and practicality.

[0122] The collaborative display method provided in the embodiments of the present application can be applied to terminal devices, wherein the terminal devices can be mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), desktop computers, smart TVs and other terminal devices with display screens.

[0123] See also Figure 1 , Figure 1 1 shows a schematic diagram of the structure of the terminal device 100. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0124] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0125] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0126] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0127] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0128] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0129] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the terminal device 100.

[0130] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0131] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0132] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0133] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 100.

[0134] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0135] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other terminal devices, such as AR devices, etc.

[0136] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0137] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 100. While the charging management module 140 is charging the battery 142, it may also power the terminal device through the power management module 141.

[0138] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0139] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0140] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0141] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0142] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0143] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0144] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0145] The terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0146] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0147] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0148] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0149] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0150] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0151] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0152] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0153] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0154] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0155] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0156] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0157] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0158] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0159] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

[0160] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0161] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the terminal device 100 detects the touch operation intensity according to the pressure sensor 180A. The terminal device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0162] The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0163] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0164] The magnetic sensor 180D includes a Hall sensor. The terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the terminal device 100 is a flip phone, the terminal device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features are set.

[0165] The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal device 100 in various directions (generally three axes). When the terminal device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the terminal device and applied to applications such as horizontal and vertical screen switching and pedometers.

[0166] The distance sensor 180F is used to measure the distance. The terminal device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0167] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light outward through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can use the proximity light sensor 180G to detect that the user holds the terminal device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, and pocket mode automatically unlocks and locks the screen.

[0168] The ambient light sensor 180L is used to sense the ambient light brightness. The terminal device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in a pocket to prevent accidental touch.

[0169] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0170] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 heats the battery 142 to avoid abnormal shutdown of the terminal device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0171] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal device 100, which is different from the position of the display screen 194.

[0172] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0173] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.

[0174] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0175] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0176] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the terminal device 100 by inserting the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0177] The software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The present application embodiment takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device 100.

[0178] Figure 2 It is a software structure block diagram of the terminal device 100 according to an embodiment of the present application.

[0179] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0180] The application layer can include a series of application packages.

[0181] like Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0182] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0183] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0184] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0185] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0186] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0187] The phone manager is used to provide communication functions of the terminal device 100, such as management of call status (including connection, disconnection, etc.).

[0188] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0189] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, the terminal device vibrates, the indicator light flashes, etc.

[0190] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0191] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0192] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0193] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0194] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0195] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0196] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0197] A 2D graphics engine is a drawing engine for 2D drawings.

[0198] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0199] The following is an illustrative description of the software and hardware workflow of the terminal device 100 in conjunction with the capture and photo shooting scene.

[0200] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0201] The collaborative display method provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0202] like Figure 3As shown, the collaborative display method provided in the embodiment of the present application can be applied to a communication system 300, and the communication system 300 may include a first terminal device 301 and a second terminal device 302. The first terminal device 301 and the second terminal device 302 can be terminal devices with display screens such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, AR / VR devices, laptops, UMPCs, netbooks, PDAs, desktop computers, and smart TVs. Among them, the size of the display screen of the second terminal device 302 is larger than the size of the display screen of the first terminal device 301, so that the second terminal device 302 can collaboratively display multiple applications in the first terminal device 301 at the same time.

[0203] When performing collaborative display, it is necessary to establish a connection between the first terminal device 301 and the second terminal device 302 to implement data interaction between the first terminal device 301 and the second terminal device 302, so that the second terminal device 302 can collaboratively display the application in the first terminal device 301. Exemplarily, the first terminal device 301 and the second terminal device 302 can establish a connection by touching, approaching automatic discovery, scanning, etc.

[0204] See also Figure 4 and Figure 5 , Figure 4 and Figure 5 Schematic diagram of establishing a connection between a first terminal device and a second terminal device is shown. In the touch connection, the first terminal device 301 and the second terminal device 302 are both terminal devices with NFC chips. When the NFC functions of the first terminal device 301 and the second terminal device 302 are both turned on, the user can use the first preset area where the NFC chip in the first terminal device 301 is located to touch the second preset area where the NFC chip in the second terminal device 302 is located, such as Figure 4 As shown, at this time, a connection pop-up box 400 may pop up in the first terminal device 301, asking whether to establish a connection with the second terminal device 302. A "Connect" button 401 and a "Cancel" button 402 may also be displayed in the connection pop-up box 400. The "Connect" button 401 may be used to send a connection request to the second terminal device 302. The "Cancel" button 402 may be used to cancel the connection operation between the first terminal device 301 and the second terminal device 302. When the user clicks or touches the "Connect" button 401 in the first terminal device 301, the first terminal device 301 may send a connection request to the second terminal device 302. Figure 5As shown in (a) in the figure, a confirmation pop-up box 500 may pop up in the second terminal device 302 to ask whether to allow the connection of the first terminal device 301. The confirmation pop-up box 500 may also display an "Allow" button 501 and a "Reject" button 502. The "Allow" button 501 can be used to establish a connection with the first terminal device 301. The "Reject" button 502 can be used to reject the connection with the first terminal device 301. When the user clicks or touches the "Allow" button 501 in the second terminal device 302, the second terminal device 302 confirms to establish a connection with the first terminal device 301. After the first terminal device 301 is connected to the second terminal device 302, as shown in FIG. Figure 5 As shown in (b), the display interface of the second terminal device 302 can display the content displayed in the first terminal device 301, for example, the main interface of the first terminal device 301 is displayed, and the main interface may include application icons of clock, calendar, gallery, memo, file management, email, music, calculator, Huawei video, sports health, weather, camera, information and other application programs. In addition, the user can operate the first terminal device 301 in the second terminal device 302 in coordination, for example, click or touch an application icon of the first terminal device 301 in the display interface of the second terminal device 302 to open the application of the first terminal device 301.

[0205] Similarly, in the close proximity automatic discovery connection, when the distance between the first terminal device 301 and the second terminal device 302 is less than the preset distance, the above Figure 4 and Figure 5 The operation in the first terminal device 301 establishes a connection between the first terminal device 301 and the second terminal device 302. Similarly, in the scan connection, after the first terminal device 301 scans the QR code for connection in the second terminal device 302, the above Figure 4 and Figure 5 The operation in establishes a connection between the first terminal device 301 and the second terminal device 302.

[0206] In one example, after the user turns on the multi-screen collaboration of the first terminal device 301 to establish a connection between the first terminal device 301 and the second terminal device 302, the first terminal device 301 can determine the first display area of ​​the main screen of the first terminal device 301 in the display interface of the second terminal device 302, and can send the display information of the main screen and the position information of the first display area to the second terminal device 302. The second terminal device 302 can then display the main screen of the first terminal device 301 in the first display area of ​​the display interface of the second terminal device 302. That is, the first terminal device 301 can project the main screen to the first display area in the second terminal device 302. Among them, the main screen is the screen on which the first terminal device 301 displays content. In other words, the first display area in the display interface of the second terminal device 302 can synchronously display the content currently displayed on the display interface of the first terminal device 301, for example, the main interface of the first terminal device 301 can be synchronously displayed.

[0207] When a user opens an application (e.g., Huawei video application) of the first terminal device 301 in the second terminal device 302, the second terminal device 302 can send the user's instruction to open the Huawei video application to the first terminal device 301. After receiving the instruction, the first terminal device 301 can create a virtual screen and determine the second display area of ​​the virtual screen in the display interface of the second terminal device 302. At the same time, the first terminal device 301 can obtain the application data of the Huawei video application, render the application data of the Huawei video application, obtain the window data, and synthesize the window data and display it on the virtual screen, that is, the Huawei video application can be opened in the virtual screen of the first terminal device 301. Subsequently, the first terminal device 301 can send the display information of the virtual screen and the position information of the second display area to the second terminal device 302. The second terminal device 302 can display the virtual screen in the second display area in the display interface of the second terminal device 302. For example, the application interface of the Huawei video screen application can be displayed in the second display area in the display interface of the second terminal device 302. In other words, the first terminal device 301 can project the virtual screen to the second display area in the second terminal device 302.

[0208] In another example, after the user turns on the multi-screen collaboration of the first terminal device 301 to establish a connection between the first terminal device 301 and the second terminal device 302, the first terminal device 301 can send the display information of the main screen to the second terminal device 302. The second terminal device 302 can determine the first display area of ​​the main screen of the first terminal device 301 in the display interface of the second terminal device 302, and then can display the main screen in the first display area of ​​the display interface of the second terminal device 302.

[0209] When a user opens an application of the first terminal device 301 in the second terminal device 302, the second terminal device 302 can send the user's instruction to open the application to the first terminal device 301. After receiving the instruction, the first terminal device 301 can create a virtual screen, obtain the application data of the application, render the application data of the application, obtain the window data, and synthesize the window data and display it on the virtual screen, that is, open the application in the virtual screen of the first terminal device 301. Subsequently, the first terminal device 301 can send the display information of the virtual screen to the second terminal device 302. The second terminal device 302 can determine the second display area of ​​the virtual screen of the first terminal device 301 in the display interface of the second terminal device 302, and then display the virtual screen in the second display area of ​​the display interface of the second terminal device 302, that is, the application interface of the application can be displayed in the second display area of ​​the display interface of the second terminal device 302.

[0210] Among them, the content displayed in the virtual screen may not be displayed in the main screen of the first terminal device 301, that is, the virtual screen may not be displayed in the first terminal device 301. The first terminal device 301 can create a virtual screen according to the current screen parameters of the main screen, and the screen parameters may include screen size and display state of the screen. The current screen parameters of the main screen refer to the screen parameters of the main screen when the virtual screen is created. However, the virtual screen is independent of the main screen, that is, although the virtual screen is created according to the screen parameters of the main screen, the screen parameters of the virtual screen are independent of the screen parameters of the main screen, that is, the screen parameters of the virtual screen will not change with the change of the screen parameters of the main screen. For example, after creating a virtual screen in a vertical screen display state according to the main screen in a vertical screen display state, when the main screen changes from a vertical screen display state to a horizontal screen display state, the screen parameters of the virtual screen can still maintain its current screen parameters, that is, the virtual screen can still maintain its current vertical screen display state and will not change to a horizontal screen display state.

[0211] It should be noted that there is no overlapping relationship between the first display area and the second display area, so as to ensure that the main screen and the virtual screen will not block each other when displayed in the second terminal device 302, thereby improving the display effect of the collaborative display of the second terminal device 302 and enhancing the user experience.

[0212] As can be seen from the above description, in the embodiment of the present application, the determination of the first display area and the second display area can be performed by the first terminal device 301, or can be performed by the second terminal device 302. Among them, the principle of the first terminal device 301 determining the first display area and the second display area is basically the same as the principle of the second terminal device 302 determining the first display area and the second display area. The following will take the determination of the first display area and the second display area by the first terminal device 301 as an example for description.

[0213] In the embodiment of the present application, the first terminal device 301 can determine the first display area and the second display area according to the display state of the first terminal device 301 and the interface information of the display interface of the second terminal device 302. Among them, for the first terminal device 301 that cannot be folded, the display state can include a horizontal screen display state and a vertical screen display state. For the foldable first terminal device 301, the display state can also include a folded screen display state and an unfolded screen display state. The following will be described respectively: (1) when the first terminal device 301 that cannot be folded is in a horizontal screen display state, the first display area and the second display area are determined; (2) when the first terminal device 301 that cannot be folded is in a vertical screen display state, the first display area and the second display area are determined; (3) when the foldable first terminal device 301 is in a folded screen display state, the first display area and the second display area are determined; (4) when the foldable first terminal device 301 is in an unfolded screen display state, the first display area and the second display area are determined.

[0214] The display state of the first terminal device 301 can be obtained by an existing method, and the embodiment of the present application does not specifically limit this. For example, the display state of the first terminal device 301 can be obtained from the activity task manager service (ATMS) in the first terminal device 301 through the window manager in the first terminal device 301. ATMS is used to manage the status of all activities in the system.

[0215] (1) When the non-foldable first terminal device 301 is in a vertical display state, determination of the first display area and the second display area

[0216] Specifically, when the first terminal device 301 that cannot be folded is in a vertical screen display state, after the first terminal device 301 establishes a connection with the second terminal device 302, the first terminal device 301 can obtain the interface information of the display interface of the second terminal device 302, and the interface information of the display interface can include the current interface height and interface width of the display interface, and the display interface of the second terminal device 302 can be vertically divided according to the current screen width of the main screen in the first terminal device 301 and the interface information of the second terminal device 302, so as to obtain the first display area and the second display area arranged left and right. Among them, the area height of the first display area is the same as the area height of the second display area, and both are the same as the current interface height of the second terminal device 302, and the sum of the area width of the first display area and the area width of the second display area is the current interface width of the second terminal device 302. Optionally, the area width of the first display area can be the same as the current screen width of the main screen in the first terminal device 301.

[0217] It should be understood that, according to the current screen width of the main screen in the first terminal device 301 and the interface information of the second terminal device 302, the display interface of the second terminal device 302 is divided vertically to obtain the first display area and the second display area arranged left and right, such as Figure 6 (a) and Figure 7 As shown in (a) in FIG. 1 , the first display area may be on the right and the second display area may be on the left; or Figure 6 (b) and Figure 7 As shown in (b), the first display area can be on the left and the second display area can be on the right. Figure 6 (a) and Figure 6 (b) is an example diagram of the first display area and the second display area when the first terminal device 301 is in a vertical screen display state and the second terminal device 302 is in a horizontal screen display state. Figure 7 (a) and Figure 7 (b) is an example diagram of the first display area and the second display area when the first terminal device 301 is in a vertical screen display state and the second terminal device 302 is in a vertical screen display state.

[0218] After the first terminal device 301 divides the first display area and the second display area, the position information of the first display area can be represented by the coordinate information of the upper left vertex and the lower right vertex of the first display area, and the position information of the second display area can be represented by the coordinate information of the upper left vertex and the lower right vertex of the second display area. The coordinate information refers to the coordinate information in the coordinate system established with the upper left vertex of the display interface of the second terminal device 302 as the coordinate origin (or the intersection between the upper edge of the display interface of the second terminal device 302 and the left edge of the display interface as the coordinate origin), the upper edge of the display interface of the second terminal device 302 as the X-axis, and the left edge of the display interface of the second terminal device 302 as the Y-axis.

[0219] Therefore, the position information of the first display area can be [(area width of the second display area, 0), (interface width of the display interface, interface height of the display interface)], and the position information of the second display area can be [(0, 0), (area width of the second display area, interface height of the display interface)]. Alternatively, the position information of the first display area can be [(0, 0), (area width of the first display area, interface height of the display interface)], and the position information of the second display area can be [(area width of the first display area, 0), (interface width of the display interface, interface height of the display interface)]. Here, the display interface refers to the display interface of the second terminal device 302.

[0220] For example, Figure 6 As shown in (a), when the first terminal device 301 is in a vertical display state and the second terminal device 302 is in a horizontal display state, the first display area can be the right area in the display interface of the second terminal device 302, and the second display area can be the left area in the display interface of the second terminal device 302, that is, the main screen and the virtual screen can be displayed horizontally in the display interface of the second terminal device 302. Among them, the area height height1 of the first display area = the area height height2 of the second display area = the interface height height of the display interface, the area width width1 of the first display area + the area width width2 of the second display area = the interface width width of the display interface, and the area width width1 of the first display area = the current screen width of the main screen in the first terminal device 301. Therefore, the position information of the first display area can be [(width2, 0), (width, height)], and the position information of the second display area can be [(0, 0), (width2, height)].

[0221] For example, Figure 7As shown in (b), when the first terminal device 301 is in a vertical screen display state and the second terminal device 302 is in a vertical screen display state, the first display area may be the left area in the display interface of the second terminal device 302, and the second display area may be the right area in the display interface of the second terminal device 302. Therefore, the position information of the first display area may be [(0, 0), (width1, height)], and the position information of the second display area may be [(width1, 0), (width, height)].

[0222] It should be noted that the use of the coordinate information of the upper left vertex and the lower right vertex of the display area (including the first display area and the second display area) to represent the position information of the display area is only for schematic interpretation and should not be understood as a limitation on the embodiments of the present application. In the embodiments of the present application, the position information of the display area can also be represented by the coordinate information of the four vertices of the display area or by the coordinate information of the lower left vertex and the upper right vertex of the display area.

[0223] (2) When the non-foldable first terminal device 301 is in a horizontal display state, determination of the first display area and the second display area

[0224] Specifically, when the first terminal device 301 that cannot be folded is in a horizontal display state, after the first terminal device 301 establishes a connection with the second terminal device 302, the first terminal device 301 can obtain the interface information of the display interface of the second terminal device 302, and the interface information of the display interface can include the current interface height and interface width of the display interface, and the display interface of the second terminal device 302 can be horizontally divided according to the current screen height of the main screen in the first terminal device 301 and the interface information of the second terminal device 302, so as to obtain the first display area and the second display area arranged vertically. Among them, the area width of the first display area is the same as the area width of the second display area, and both are the same as the current interface width of the second terminal device 302, and the sum of the area height of the first display area and the area height of the second display area is the current interface height of the second terminal device 302. Optionally, the area height of the first display area can be the same as the current screen height of the main screen in the first terminal device 301.

[0225] It should be understood that, according to the current screen height of the main screen in the first terminal device 301 and the interface information of the second terminal device 302, the display interface of the second terminal device 302 is horizontally divided to obtain the first display area and the second display area arranged vertically, such as Figure 8 (a) and Fig. 9 As shown in (a) in FIG. 1 , the first display area may be on the upper side and the second display area may be on the lower side; or Figure 8 (b) and Fig. 9 As shown in (b), the first display area can be at the bottom and the second display area can be at the top. Figure 8 (a) and Figure 8 (b) is an example diagram of the first display area and the second display area when the first terminal device 301 is in a horizontal screen display state and the second terminal device 302 is in a horizontal screen display state. Fig. 9 (a) and Fig. 9 (b) is an example diagram of the first display area and the second display area when the first terminal device 301 is in a horizontal display state and the second terminal device 302 is in a vertical display state.

[0226] Similar to (1), after the first terminal device 301 divides the first display area and the second display area, the position information of the first display area can be represented by the coordinate information of the upper left vertex and the lower right vertex of the first display area, and the position information of the second display area can be represented by the coordinate information of the upper left vertex and the lower right vertex of the second display area. Similarly, the coordinate information refers to the coordinate information in the coordinate system established with the upper left vertex of the display interface of the second terminal device 302 as the coordinate origin (or the intersection between the upper edge of the display interface of the second terminal device 302 and the left edge of the display interface as the coordinate origin), the upper edge of the display interface of the second terminal device 302 as the X-axis, and the left edge of the display interface of the second terminal device 302 as the Y-axis.

[0227] Therefore, the position information of the first display area can be [(0, 0), (interface width of the display interface, region height of the first display area)], and the position information of the second display area can be [(0, region height of the first display area), (interface width of the display interface, interface height of the display interface)]. Alternatively, the position information of the first display area can be [(0, region height of the second display area), (interface width of the display interface, interface height of the display interface)], and the position information of the second display area can be [(0, 0), (interface width of the display interface, region height of the second display area)]. Here, the display interface refers to the display interface of the second terminal device 302.

[0228] For example, Figure 8As shown in (a), when the first terminal device 301 is in a horizontal display state and the second terminal device 302 is in a horizontal display state, the first display area can be the upper area in the display interface of the second terminal device 302, and the second display area can be the lower area in the display interface of the second terminal device 302, that is, the main screen and the virtual screen can be displayed vertically arranged up and down in the display interface of the second terminal device 302. Among them, the area width width1 of the first display area = the area width width2 of the second display area = the interface width width of the display interface, the area height height1 of the first display area + the area height height2 of the second display area = the interface height height of the display interface, and the area height height1 of the first display area = the current screen height of the main screen in the first terminal device 301. Therefore, the position information of the first display area can be [(0, 0), (width, height1)], and the position information of the second display area can be [(0, height1), (width, height)].

[0229] For example, Fig. 9 As shown in (b), when the first terminal device 301 is in a horizontal display state and the second terminal device 302 is in a vertical display state, the first display area may be the lower area in the display interface of the second terminal device 302, and the second display area may be the upper area in the display interface of the second terminal device 302. Therefore, the position information of the first display area may be [(0, height2), (width, height)], and the position information of the second display area may be [(0, 0), (width, height2)].

[0230] (3) Determination of the first display area and the second display area when the foldable first terminal device 301 is in the folded screen display state

[0231] It should be understood that in a scenario where the foldable first terminal device 301 is in a folded screen display state, when the folded first terminal device 301 is in a vertical screen display state, the first terminal device 301 can determine the first display area and the second display area according to the method in (1) above. That is, the first terminal device 301 can vertically divide the display interface of the second terminal device 302 according to the current screen width of the main screen in the first terminal device 301 and the interface information of the second terminal device 302, to obtain the first display area and the second display area arranged left and right.

[0232] When the folded first terminal device 301 is in a horizontal display state, the first terminal device 301 can determine the first display area and the second display area according to the method in (2) above. That is, the first terminal device 301 can divide the display interface of the second terminal device 302 horizontally according to the current screen height of the main screen of the first terminal device 301 and the interface information of the second terminal device 302 to obtain the first display area and the second display area arranged vertically.

[0233] (4) Determination of the first display area and the second display area when the foldable first terminal device 301 is in the unfolded screen display state

[0234] Specifically, when the foldable first terminal device 301 is in the expanded screen display state, the main screen of the first terminal device 301 will become larger accordingly, and the area of ​​the first display area also needs to be increased accordingly. In order to make full use of the display interface of the second terminal device 302 and improve the utilization rate of the display interface in the second terminal device 302, the first terminal device 301 can determine the first display area and the second display area in combination with the display state of the second terminal device 302.

[0235] The display state of the second terminal device 302 can be determined according to the interface information of the display interface in the second terminal device 302. Exemplarily, when the current screen width of the display interface is greater than the current screen height, it can be determined that the second terminal device 302 is in a horizontal screen display state; when the current screen width of the display interface is less than the current screen height, it can be determined that the second terminal device 302 is in a vertical screen display state.

[0236] In one example, when the second terminal device 302 is in a horizontal display state, the first terminal device 301 can determine the first display area and the second display area by referring to the method in (1) above. That is, the first terminal device 301 can vertically divide the display interface of the second terminal device 302 according to the current screen width of the main screen in the first terminal device 301 and the interface information of the second terminal device 302, and obtain the first display area and the second display area arranged on the left and right, so as to ensure the display area of ​​the second display area, so that the second display area can display more virtual screens, improve the utilization rate of the display interface of the second terminal device 302, and enhance the user experience. Similar to (1), in the vertical division of the display interface of the second terminal device 302 according to the current screen width of the main screen in the first terminal device 301 and the interface information of the second terminal device 302, and obtain the first display area and the second display area arranged on the left and right, the first display area can be on the left and the second display area can be on the right; or the first display area can be on the right and the second display area can be on the left.

[0237] Therefore, the position information of the first display area can be [(0, 0), (area width of the first display area, interface height of the display interface)], and the position information of the second display area can be [(area width of the first display area, 0), (interface width of the display interface, interface height of the display interface)]. Alternatively, the position information of the first display area can be [(area width of the second display area, 0), (interface width of the display interface, interface height of the display interface)], and the position information of the second display area can be [(0, 0), (area width of the second display area, interface height of the display interface)].

[0238] In another example, when the second terminal device 302 is in a vertical screen display state, the first terminal device 301 can determine the first display area and the second display area by referring to the method in (2) above. That is, the first terminal device 301 can divide the display interface of the second terminal device 302 horizontally according to the current screen height of the main screen in the first terminal device 301 and the interface information of the second terminal device 302, and obtain the first display area and the second display area arranged vertically, so as to ensure the display area of ​​the second display area, so that the second display area can display more virtual screens, improve the utilization rate of the display interface of the second terminal device 302, and enhance the user experience. Similar to (2), in the case where the display interface of the second terminal device 302 is divided horizontally according to the current screen height of the main screen in the first terminal device 301 and the interface information of the second terminal device 302, and the first display area and the second display area arranged vertically are obtained, the first display area can be on the upper side of the display interface and the second display area can be on the lower side of the display interface; or the first display area can be on the lower side of the display interface and the second display area can be on the upper side of the display interface.

[0239] Therefore, the position information of the first display area can be [(0, 0), (interface width of the display interface, region height of the first display area)], and the position information of the second display area can be [(0, region height of the first display area), (interface width of the display interface, interface height of the display interface)]. Alternatively, the position information of the first display area can be [(0, region height of the second display area), (interface width of the display interface, interface height of the display interface)], and the position information of the second display area can be [(0, 0), (interface width of the display interface, region height of the second display area)].

[0240] In a possible implementation, in a scenario where the first terminal device 301 is in an expanded screen display state, when the difference between the screen width of the main screen and the screen height of the main screen is less than a preset threshold, the first terminal device 301 can directly determine the area width of the first display area according to the larger of the screen width of the main screen and the screen height of the main screen. That is, when the first terminal device 301 is in a horizontal screen display state, the first terminal device 301 can determine the larger value of the current screen height of the main screen and the current screen height (for example, the current screen height of the main screen) as the area width of the first display area; when the first terminal device 301 is in a vertical screen display state, the first display area and the second display area remain the same as the first display area and the second display area when the first terminal device 301 is in a horizontal screen display state. Therefore, when the first terminal device 301 changes from a horizontal screen display state to a vertical screen display state, it is not necessary to reorganize the first display area and the second display area, thereby reducing the operation delay and improving the display effect. Among them, the preset threshold can be specifically set according to actual conditions.

[0241] In the embodiment of the present application, after determining the first display area and the second display area, the second terminal device 302 can display the main screen in the first display area and can display the virtual screen in the second display area. It should be understood that when displaying the main screen in the first display area, it is necessary to first determine the relative position of the main screen in the first display area, so that the second terminal device 302 can display the main screen in the first display area according to the relative position.

[0242] The relative position of the main screen in the first display area may be set by default by the first terminal device 301 or the second terminal device 302, or may be set by a user in the first terminal device 301 or the second terminal device 302. When the relative position of the main screen in the first display area is set by default by the first terminal device 301 or is set by a user in the first terminal device 301, the first terminal device 301 may send the relative position of the main screen in the first display area to the second terminal device 302 at the same time when sending the display information of the main screen, or sending the display information of the main screen and the position information of the first display area to the second terminal device 302.

[0243] Similarly, when displaying a virtual screen in the second display area, it is necessary to first determine the relative position of the virtual screen in the second display area, so that the second terminal device 302 can display the virtual screen in the second display area according to the relative position. The relative position of the virtual screen in the second display area can be set by default by the first terminal device 301 or the second terminal device 302, or can be customized by the user in the first terminal device 301 or the second terminal device 302. When the relative position of the virtual screen in the second display area is set by default by the first terminal device 301 or customized by the user in the first terminal device 301, the first terminal device 301 can send the relative position of the virtual screen in the second display area when sending the display information of the virtual screen to the second terminal device 302, or sending the display information of the virtual screen and the position information of the second display area.

[0244] In a possible implementation, the relative position of the virtual screen in the second display area can also be determined according to the arrangement order of the virtual screens in the second display area. The arrangement order can be set by default by the first terminal device 301 or the second terminal device 302, or can be customized by the user in the first terminal device 301 or the second terminal device 302. For example, the second terminal device 302 can be set by default to display the virtual screens in an arrangement order from left to right. Therefore, when the second terminal device 302 receives the display information of the first virtual screen sent by the first terminal device 301, the second terminal device 302 can display the first virtual screen at the leftmost in the second display area; subsequently, when the second terminal device 302 receives the display information of the second virtual screen sent by the first terminal device 301, the second terminal device 302 can display the second virtual screen to the right of the first virtual screen, and so on, until no more virtual screens can be displayed in the second display area, or until the number of virtual screens displayed in the second display area reaches a preset maximum number. The preset maximum number can be customized by the user.

[0245] In the embodiment of the present application, in the collaborative display through the second terminal device 302, when the display state of the first terminal device 301 changes, the state of the main screen displayed in the second terminal device 302 will also change. Therefore, in order to improve the display effect of the main screen in the second terminal device 302 and improve the user experience, the first display area and the second display area can be reorganized, and the main screen is displayed in the reorganized first display area and the virtual screen is displayed in the reorganized second display area. Among them, the reorganization of the first display area and the second display area can be performed by the first terminal device 301 or the second terminal device 302. The following will be described by taking the reorganization of the first display area and the second display area by the first terminal device 301 as an example.

[0246] Specifically, when the first terminal device 301 changes from a vertical screen display state to a horizontal screen display state, the first terminal device 301 can reorganize the first display area and the second display area by referring to the method in (2) above, that is, the first display area and the second display area arranged left and right can be reorganized into the first display area and the second display area arranged up and down. The area width of the first display area before reorganization is equal to the area height of the first display area after reorganization. For example, the first terminal device 301 can Fig.10 The first display area and the second display area shown in (a) are arranged left and right, and are reorganized into Fig.10 The first display area and the second display area are arranged vertically as shown in (b) in FIG. Fig.10 The area height height1 of the first display area shown in (b) is equal to Fig.10 The area width width1 of the first display area shown in (a) is shown in FIG.

[0247] When the first terminal device 301 changes from a horizontal screen display state to a vertical screen display state, the first terminal device 301 can reorganize the first display area and the second display area by referring to the method in (1) above, that is, reorganize the first display area and the second display area arranged vertically into the first display area and the second display area arranged horizontally. The area height of the first display area before reorganization is equal to the area width of the first display area after reorganization. For example, the first terminal device 301 can Fig.11 The first display area and the second display area arranged vertically as shown in (a) are reorganized into Fig.11 The first display area and the second display area are arranged left and right as shown in (b) in FIG. Fig.11 The area width width1 of the first display area shown in (b) is equal to Fig.11 The area height height1 of the first display area shown in (a) is shown in FIG.

[0248] When the first terminal device 301 changes from the folded screen display state to the unfolded screen display state, the first terminal device 301 can reorganize the first display area and the second display area in accordance with the aforementioned method (4). That is, when the second terminal device 302 is in the horizontal screen display state, the first terminal device 301 can vertically divide the display interface of the second terminal device 302 according to the screen width of the unfolded main screen and the interface information of the second terminal device 302, so as to obtain the first display area and the second display area arranged left and right. When the second terminal device 302 is in the vertical screen display state, the first terminal device 301 can horizontally divide the display interface of the second terminal device 302 according to the screen height of the unfolded main screen and the interface information of the second terminal device 302, so as to obtain the first display area and the second display area arranged up and down.

[0249] For example, when the second terminal device 302 is in a horizontal screen display state, the first terminal device 301 can Fig.12 The first display area and the second display area shown in (a) are reorganized into Fig.12 The first display area and the second display area shown in (b) of FIG. The area width1 of the first display area before the reorganization is equal to the screen width of the main screen of the first terminal device 301 after folding, and the area width1' of the first display area after the reorganization is equal to the screen width of the main screen of the first terminal device 301 after unfolding. For example, when the second terminal device 302 is in the vertical screen display state, the first terminal device 301 can Fig.13 The first display area and the second display area shown in (a) are reorganized into Fig.13 The first display area and the second display area shown in (b) of FIG. The area width width1 of the first display area before reorganization is equal to the screen width of the main screen of the first terminal device 301 after folding, and the area height height1 of the first display area after reorganization is equal to the screen height of the main screen of the first terminal device 301 after unfolding.

[0250] In a possible implementation, in a scenario where the first terminal device 301 is in an expanded screen display state, when the first terminal device 301 changes from a horizontal screen display state to a vertical screen display state, the first terminal device 301 may reorganize the first display area and the second display area by referring to the method described in (1) above; when the first terminal device 301 changes from a vertical screen display state to a horizontal screen display state, the first terminal device 301 may reorganize the first display area and the second display area by referring to the method described in (2) above.

[0251] In another possible implementation, in a scenario where the first terminal device 301 is in an expanded screen display state, when the first terminal device 301 determines the first display area and the second display area based on the larger of the screen width and the screen height of the expanded main screen, for example, when the expanded first terminal device 301 is in a horizontal screen display state, if the first terminal device 301 determines the area width of the first display area based on the screen height of the larger main screen, then when the first terminal device 301 changes from a horizontal screen display state to a vertical screen display state, the first terminal device 301 does not need to reorganize the first display area and the second display area, and can directly display the content of the first terminal device 301 in the vertical screen display state with the first display area and the second display area determined in the horizontal screen display state, thereby reducing the operation delay of the collaborative display and improving the display effect.

[0252] When the first terminal device 301 changes from the unfolded screen display state to the folded screen display state, the first terminal device 301 can reorganize the first display area and the second display area in accordance with the aforementioned method (3). That is, when the first terminal device 301 is in the horizontal screen display state, the first terminal device 301 can divide the display interface of the second terminal device 302 horizontally according to the screen height of the main screen of the folded first terminal device 301 and the interface information of the second terminal device 302, so as to obtain the first display area and the second display area arranged vertically. When the first terminal device 301 is in the vertical screen display state, the first terminal device 301 can divide the display interface of the second terminal device 302 vertically according to the screen width of the main screen of the folded first terminal device 301 and the interface information of the second terminal device 302, so as to obtain the first display area and the second display area arranged horizontally.

[0253] For example, when the unfolded first terminal device 301 and the folded first terminal device 301 are in a horizontal display state, the first terminal device 301 can Fig.14 The first display area and the second display area shown in (a) are reorganized into Fig.14 The first display area and the second display area shown in (b) of FIG. The area height height1 of the first display area before the reorganization is equal to the screen height of the main screen of the first terminal device 301 after the first terminal device 301 is unfolded, and the area height height1' of the first display area after the reorganization is equal to the screen height of the main screen of the first terminal device 301 after the first terminal device 301 is folded. For example, when the unfolded first terminal device 301 and the folded first terminal device 301 are in the vertical screen display state, the first terminal device 301 can Fig.14 The first display area and the second display area shown in (c) are reorganized into Fig.14The first display area and the second display area shown in (d) in FIG. 1 . The area height height1 of the first display area before the reorganization is equal to the screen height of the main screen after the first terminal device 301 is unfolded, and the area width width1 of the first display area after the reorganization is equal to the screen width of the main screen after the first terminal device 301 is folded.

[0254] Optionally, when the display state of the first terminal device 301 changes, but the content display mode of the main screen in the first terminal device 301 does not change accordingly, if the first display area and the second display area are reorganized, it will affect the display effect of the content of the main screen in the second terminal device 302, making it inconvenient for the user to browse. Therefore, when the display state of the first terminal device 301 changes, if the content display mode of the main screen changes with the change of the display state of the first terminal device 301, the first terminal device 301 can reorganize the first display area and the second display area; if the content display mode of the main screen does not change with the change of the display state of the first terminal device 301, the first terminal device 301 may not reorganize the first display area and the second display area.

[0255] The content display mode of the main screen can be determined by existing methods, and the embodiments of the present application do not specifically limit this. For example, the horizontal and vertical states of the activity corresponding to the main screen can be obtained through the activity manager service (AMS) in the first terminal device 301, so as to determine the content display mode of the main screen according to the horizontal and vertical states of the activity.

[0256] Optionally, when the display state of the first terminal device 301 has not changed, but the content display mode of the main screen in the first terminal device 301 has changed, for example, when the user clicks the horizontal or vertical screen switch button of the application displayed on the main screen to change the content display mode of the main screen, but does not rotate the first terminal device 301 accordingly, in order to improve the display effect of the content of the main screen in the second terminal device 302, to facilitate the user to browse the content in the second terminal device 302 and enhance the user's browsing experience, the first terminal device 301 may also reorganize the first display area and the second display area according to the change in the content display mode.

[0257] Among them, the content display mode may include horizontal screen display and vertical screen display. In the embodiment of the present application, the reorganization of the first display area and the second display area according to the content display mode can be performed with reference to the methods in the aforementioned (1) and (2). Specifically, when the content display mode of the main screen changes from vertical screen display to horizontal screen display, the first terminal device 301 can refer to the method in the aforementioned (2) to reorganize the first display area and the second display area. That is, the first terminal device 301 can obtain the interface information of the display interface in the second terminal device 302, and can divide the display interface of the second terminal device 302 horizontally according to the current screen width of the main screen in the first terminal device 301 (that is, the screen width in the vertical screen display state) and the interface information of the second terminal device 302, so as to obtain the first display area and the second display area arranged vertically. When the content display mode of the main screen changes from horizontal screen display to vertical screen display, the first terminal device 301 can reorganize the first display area and the second display area according to the method in the aforementioned (1). That is, the first terminal device 301 can obtain the interface information of the display interface in the second terminal device 302, and can vertically divide the display interface of the second terminal device 302 according to the current screen height of the main screen in the first terminal device 301 (that is, the screen height in the horizontal display state) and the interface information of the second terminal device 302 to obtain the first display area and the second display area arranged left and right.

[0258] In the embodiment of the present application, after the first display area and the second display area are reorganized, the content display mode and relative position of the main screen in the first display area can change with the change of the display state of the first terminal device 301 or with the change of the content display mode of the main screen. The virtual screen in the second display area still maintains the content display mode and relative position before the reorganization for display. For example, before the reorganization, if the first virtual screen and the second virtual screen of the second display area are displayed in portrait mode and arranged in the second display area from left to right, then after the first display area and the second display area are reorganized, the first virtual screen and the second virtual screen are still displayed in portrait mode and arranged in the second display area after the reorganization, so as to reduce the re-refresh and layout of the content in the virtual screen, reduce the display delay of the second display area, thereby improving the display effect and enhancing the user experience.

[0259] It is understandable that when the area height of the reorganized second display area is less than the screen height of the virtual screen, or the area width of the reorganized second display area is less than the screen width of the virtual screen, to ensure that the virtual screen can be fully displayed in the second terminal device 302, before the second terminal device 302 displays the virtual screen in the second display area, the virtual screen can be scaled down according to the reorganized second display area, and then the scaled down virtual screen can be fully displayed in the second display area of ​​the second terminal device 302 according to the relative position of the virtual screen in the second display area before the reorganization. Alternatively, the first terminal device 301 can scale down the virtual screen according to the reorganized second display area, and can send the display information of the scaled down virtual screen to the second terminal device 302. The second terminal device 302 can fully display the virtual screen in the second display area according to the relative position of the virtual screen in the second display area before the reorganization.

[0260] The collaborative display method provided in the embodiment of the present application is exemplarily described below in combination with a specific application scenario. In the following application scenario, the first terminal device 301 is used as an example to determine and reorganize the first display area and the second display area.

[0261] For example, in a scenario where the first terminal device 301 is in a portrait display state and the second terminal device 302 is in a landscape display state, after the first terminal device 301 and the second terminal device 302 perform multi-screen collaboration, the first terminal device 301 can determine that the first display area and the second display area are arranged horizontally from left to right, for example, the first display area is on the right and the second display area is on the left. The first terminal device 301 can send the display information of the main screen in the first terminal device 301 and the position information of the first display area to the second terminal device 302. Fig.15 As shown in (a), after the second terminal device 302 receives the display information of the main screen and the position information of the first display area, the main screen 1500 can be displayed in the first display area in the display interface of the second terminal device 302.

[0262] When the user clicks on or touches the SMS application 1501 in the main screen 1500 in the second terminal device 302, the second terminal device 302 can send the instruction of the user clicking or touching the SMS application to the first terminal device 301. After receiving the instruction, the first terminal device 301 can create a first virtual screen, obtain the application data of the SMS application, render the application data of the SMS application, obtain the window data, and synthesize the window data and display it in the first virtual screen. Subsequently, the first terminal device 301 can send the display information of the first virtual screen and the location information of the second display area to the second terminal device 302. Fig.15As shown in (b), after the second terminal device 302 receives the display information of the first virtual screen and the position information of the second display area, the first virtual screen can be displayed in the second display area in the display interface of the second terminal device 302 in an arrangement order from left to right, that is, the content of the SMS application can be displayed on the far left of the second display area.

[0263] When the user continues to click or touch the Huawei video application 1502 in the main screen 1500 in the second terminal device 302, the second terminal device 302 can send the instruction of the user clicking or touching the Huawei video application to the first terminal device 301. After receiving the instruction, the first terminal device 301 can create a second virtual screen, obtain the application data of the Huawei video application, render the application data of the Huawei video application, obtain the window data, and synthesize the window data and display it in the second virtual screen. Subsequently, the first terminal device 301 can send the display information of the second virtual screen to the second terminal device 302. Fig.16 As shown in (a), after the second terminal device 302 receives the display information of the second virtual screen, it can display the second virtual screen on the right side of the SMS application in the second display area in a display order from left to right.

[0264] When the user clicks or touches the gallery application in the first terminal device 301, the first terminal device 301 can open the gallery application to the main screen and send the display information of the main screen to the second terminal device 302. Fig.16 As shown in (b), after the second terminal device 302 receives the display information of the main screen, the content of the gallery application can be displayed in the first display area.

[0265] In the scenario of opening the SMS application, Huawei video application and gallery application, when the user rotates the first terminal device 301 and the content display mode of the main screen in the first terminal device 301 changes accordingly, that is, when the user switches the first terminal device 301 from the vertical screen display state to the horizontal screen display state, and the content displayed in the first terminal device 301 will also change from the vertical screen display to the horizontal screen display, the first terminal device 301 can reorganize the first display area and the second display area arranged horizontally from left to right into the first display area and the second display area arranged vertically from top to bottom, for example, the reorganized first display area is on the upper side and the second display area is on the lower side. Subsequently, the first terminal device 301 can send the display information of the rotated main screen and the position information of the reorganized first display area to the second terminal device 302, and at the same time send the position information of the reorganized second display area and the display information of the first virtual screen and the second virtual screen to the second terminal device 302. Among them, the first virtual screen and the second virtual screen will not change with the change of the display state of the first terminal device 301, and the relative positions of the first virtual screen and the second virtual screen in the second display area remain unchanged. As Fig.17 As shown, after the second terminal device 302 receives the display information of the rotated main screen, the position information of the reorganized first display area, the position information of the reorganized second display area, and the display information of the first virtual screen and the second virtual screen, the main screen can be displayed in the reorganized first display area, and the first virtual screen and the second virtual screen can be displayed in the reorganized second display area in portrait mode according to the relative positions of the first virtual screen and the second virtual screen in the second display area before the reorganization.

[0266] It is understandable that when the user rotates the first terminal device 301, if the content in the virtual screen (including at least one of the first virtual screen and the second virtual screen) has not changed, the first terminal device 301 does not need to send the display information of the virtual screen to the second terminal device 302. The second terminal device 302 can directly display the virtual screen in the reorganized second display area based on the previously acquired display information of the virtual screen and the relative position of the virtual screen in the second display area, thereby improving the display efficiency of the virtual screen.

[0267] For example, in a scenario where the first terminal device 301 is in a folded vertical screen display state and the second terminal device 302 is in a vertical screen display state, after the first terminal device 301 and the second terminal device 302 perform multi-screen collaboration, the first terminal device 301 can determine that the first display area and the second display area are arranged horizontally left and right, for example, the first display area is on the right and the second display area is on the left. The first terminal device 301 can send the display information of the main screen 1800 in the first terminal device 301 and the position information of the first display area to the second terminal device 302. Fig.18 As shown, after the second terminal device 302 receives the display information of the main screen 1800 and the position information of the first display area, the main screen 1800 can be displayed in the first display area in the display interface of the second terminal device 302.

[0268] When the user clicks on the Huawei video application 1801 in the main screen 1800 in the second terminal device 302, the second terminal device 302 can send the instruction of the user clicking on the Huawei video application to the first terminal device 301. After receiving the instruction, the first terminal device 301 can create a first virtual screen, obtain the application data of the Huawei video application, render the application data of the Huawei video application, obtain the window data, and synthesize the window data and display it on the first virtual screen. Subsequently, the first terminal device 301 can send the display information of the first virtual screen and the location information of the second display area to the second terminal device 302. Fig.19 As shown, after the second terminal device 302 receives the display information of the first virtual screen and the location information of the second display area, it can display the first virtual screen in the second display area in the display interface of the second terminal device 302 in a display order from right to left, that is, display the Huawei video application on the right side of the second display area.

[0269] When the user clicks or touches the SMS application in the first terminal device 301, the first terminal device 301 can open the SMS application to the main screen and send the display information of the main screen to the second terminal device 302. Fig. 20 As shown, after receiving the display information of the main screen, the second terminal device 302 can display the content of the SMS application in the first display area.

[0270] In the scenario of opening the SMS application and the Huawei video application, when the user expands the first terminal device 301, the first terminal device 301 can reorganize the first display area and the second display area that are arranged horizontally from left to right into the first display area and the second display area that are arranged vertically from top to bottom. For example, the reorganized first display area is on the upper side and the second display area is on the lower side. Subsequently, the first terminal device 301 can send the display information of the expanded main screen and the position information of the reorganized first display area to the second terminal device 302, and at the same time send the position information of the reorganized second display area and the display information of the first virtual screen to the second terminal device 302. Among them, the first virtual screen will not change with the change of the display state of the first terminal device 301, and the relative position of the first virtual screen in the second display area remains unchanged. Fig.21 As shown, after the second terminal device 302 receives the display information of the expanded main screen, the position information of the reorganized first display area, the position information of the reorganized second display area and the display information of the first virtual screen, the main screen can be displayed in the reorganized first display area, and the first virtual screen can be displayed in the reorganized second display area in a vertical display mode according to the relative position of the first virtual screen in the second display area before the reorganization, that is, the first virtual screen is still displayed in a vertical display mode on the right side of the reorganized second display area.

[0271] It is understandable that when the user rotates the first terminal device 301, if the content in the first virtual screen has not changed, the first terminal device 301 does not need to send the display information of the first virtual screen to the second terminal device 302. The second terminal device 302 can directly display the first virtual screen in the reorganized second display area according to the previously acquired display information of the first virtual screen and the relative position of the first virtual screen in the second display area, thereby improving the display efficiency of the virtual screen.

[0272] Based on the above description, the collaborative display method provided by the embodiment of the present application is briefly described below. Fig. 22 , Fig. 22 FIG. 1 shows a schematic flow chart of a collaborative display method provided in an embodiment of the present application. Fig. 22 As shown, the method may include:

[0273] 1. The user opens multi-screen collaboration on the first terminal device;

[0274] 2. The first terminal device establishes a connection with the second terminal device;

[0275] It is understandable that the user can use the first preset area where the NFC chip in the first terminal device is located to touch the second preset area where the NFC chip in the second terminal device is located to enable multi-screen collaboration. At this time, a connection pop-up box may pop up in the first terminal device to ask whether to establish a connection with the second terminal device. The connection pop-up box may include a "Connect" button and a "Cancel" button. When the user clicks the "Connect" button in the first terminal device, the first terminal device may send a connection request to the second terminal device. After the second terminal device receives the connection request, a confirmation pop-up box may pop up to ask whether to allow the connection of the first terminal device. The confirmation pop-up box may include an "Allow" button and a "Reject" button. When the user clicks the "Allow" button in the second terminal device, a connection is established between the second terminal device and the first terminal device, and the second terminal device can collaboratively display the content of the first terminal device.

[0276] 3. The first terminal device determines a first display area of ​​the main screen in the second terminal device according to the first display state of the first terminal device, and sends display information of the main screen and first position information of the first display area to the second terminal device;

[0277] 4. The second terminal device displays the main screen in the first display area;

[0278] 5. The user clicks on the second terminal device to open the application in the first terminal device;

[0279] 6. The second terminal device sends an instruction to open the application to the first terminal device;

[0280] 7. After receiving the instruction, the first terminal device creates a virtual screen, displays the application on the virtual screen, determines a second display area of ​​the virtual screen in the second terminal device, and sends display information of the virtual screen and second position information of the second display area to the second terminal device;

[0281] Specifically, the first terminal device can obtain the first display state of the first terminal device and the interface information of the display interface of the second terminal device, and can determine the first display area of ​​the main screen in the display interface of the second terminal device and the second display area of ​​the virtual screen in the display interface of the second terminal device based on the first display state of the first terminal device and the interface information of the second terminal device.

[0282] The display state of the first terminal device 301 can be obtained by existing methods, and the embodiment of the present application does not specifically limit this. For example, the display state of the first terminal device 301 can be obtained from the active ATMS in the first terminal device 301 through the window manager in the first terminal device 301. In addition, after determining the first display area corresponding to the main screen and the second display area corresponding to the virtual screen, the window manager can obtain and save the position information of the first display area and the position information of the second display area.

[0283] For example, when the first terminal device is a non-foldable terminal device and the first terminal device is in a portrait display state, or when the first terminal device is a foldable terminal device and the first terminal device is in a portrait display state after being folded, the first terminal device can determine that the first display area and the second display area are arranged horizontally left and right, and the area heights of the first display area and the second display area are the same as the interface height of the display interface of the second terminal device, and the sum of the area widths of the first display area and the second display area is equal to the interface width of the display interface of the second terminal device.

[0284] For example, when the first terminal device is a non-foldable terminal device and the first terminal device is in a horizontal screen display state, or when the first terminal device is a foldable terminal device and the first terminal device is in a horizontal screen display state after being folded, the first terminal device can determine that the first display area and the second display area are arranged vertically up and down, and the area widths of the first display area and the second display area are the same as the interface width of the display interface of the second terminal device, and the sum of the area heights of the first display area and the second display area is equal to the interface height of the display interface of the second terminal device.

[0285] For example, when the first terminal device is a foldable terminal device and the first terminal device is in an unfolded screen display state, the first terminal device can determine the first display area and the second display area according to the interface information of the second terminal device. When it is determined according to the interface information of the second terminal device that the second terminal device is in a horizontal screen display state, the first terminal device can determine that the first display area and the second display area are arranged horizontally from left to right, and the area heights of the first display area and the second display area are the same as the interface height of the display interface of the second terminal device, and the sum of the area widths of the first display area and the second display area is equal to the interface width of the display interface of the second terminal device. When it is determined according to the interface information of the second terminal device that the second terminal device is in a vertical screen display state, the first terminal device can determine that the first display area and the second display area are arranged vertically from top to bottom, and the area widths of the first display area and the second display area are the same as the interface width of the display interface of the second terminal device, and the sum of the area heights of the first display area and the second display area is equal to the interface height of the display interface of the second terminal device.

[0286] 8. The second terminal device displays a virtual screen in the second display area;

[0287] 9. The user adjusts the first terminal device from the first display state to the second display state;

[0288] 10. The first terminal device reorganizes the first display area and the second display area according to the second display state, and sends the display information of the main screen, the third position information of the reorganized first display area, the display information of the virtual screen, and the fourth position information of the reorganized second display area to the second terminal device;

[0289] In the embodiment of the present application, the user can adjust the first terminal device from the first display state to the second display state by rotating the first terminal device. The second display state is any one of a portrait display state, a landscape display state, a folded screen display state, and an expanded screen display state. For example, the first terminal device can be adjusted from a portrait display state to a landscape display state; or from a landscape display state to a portrait display state; or from a folded screen display state to an expanded screen display state; or from an expanded screen display state to a folded screen display state, and so on.

[0290] When the display state of the first terminal device changes, the configuration file of the AMS in the first terminal device will change accordingly. The AMS can send a message to the window manager in the first terminal device that the display state of the first terminal device has changed. After receiving the message, the window manager can reacquire and save the position information of the first display area and the position information of the second display area.

[0291] After the first display area and the second display area are reorganized, the first terminal device can transmit the display information of the main screen in the first terminal device, the third position information of the reorganized first display area, the display information of the virtual screen and the fourth position information of the reorganized second display area to the second terminal device, so that the second terminal device can display the main screen and the virtual screen of the first terminal device.

[0292] 11. The second terminal device displays the main screen in the reorganized first display area, and displays the virtual screen in the reorganized second display area.

[0293] The content display mode of the main screen may change with the display state of the first terminal device, but the content display mode of the virtual screen will remain unchanged, that is, the content display mode of the virtual screen will not change with the display state of the first terminal device. For example, when both the main screen and the virtual screen are displayed in portrait mode, when the first terminal device changes from the portrait mode to the landscape mode, the main screen may change from the portrait mode to the landscape mode, but the virtual screen may still remain in the portrait mode.

[0294] In addition, the relative position of the virtual screen in the second display area after the reorganization is the same as the relative position of the virtual screen in the second display area before the reorganization, that is, when the display state of the first terminal device changes, there is no need to refresh and layout the content of the application displayed on the virtual screen, that is, the second terminal device can continue to display the virtual screen according to the relative position of the virtual screen in the second display area before the reorganization and the original content display method of the virtual screen.

[0295] In the embodiment of the present application, the first display area of ​​the main screen of the first terminal device in the second terminal device and the second display area of ​​the virtual screen in the second terminal device can be determined according to the display state of the first terminal device, so that the second terminal device can display the main screen in the first display area and the virtual screen in the second display area. When the display state of the first terminal device changes, only the content display mode of the main screen and the first display area and the second display area can be dynamically adjusted without adjusting the content display mode of the virtual screen and the relative position of the virtual screen in the second display area, so that there is no need to re-layout and refresh the content of the application displayed on the virtual screen, reduce the display delay of the content displayed in the second display area, improve user experience, and have strong ease of use and practicality.

[0296] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0297] The embodiment of the present application also provides a terminal device, the terminal device includes at least one memory, at least one processor, and a computer program stored in the at least one memory and executable on the at least one processor, and when the processor executes the computer program, the terminal device implements the steps in any of the above-mentioned method embodiments. Exemplarily, the structure of the terminal device can be as follows: Figure 1 shown.

[0298] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a computer, the computer implements the steps in any of the above method embodiments.

[0299] An embodiment of the present application provides a computer program product. When the computer program product is executed on a terminal device, the terminal device implements the steps in any of the above method embodiments.

[0300] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, a computer-readable storage medium cannot be an electric carrier signal and a telecommunication signal.

[0301] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0302] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0303] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0304] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0305] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A collaborative display method, applied to a first terminal device, characterized in that: The method comprises: Determine, according to the first display state of the first terminal device, a first display area of ​​the first main screen of the first terminal device in the second terminal device and a second display area of ​​the virtual screen of the first terminal device in the second terminal device; Casting the first main screen to the second terminal device, and sending first position information of the first display area to the second terminal device, where the first position information is used by the second terminal device to display the first main screen in the first display area; The virtual screen is projected to the second terminal device, and second position information of the second display area is sent to the second terminal device, where the second position information is used by the second terminal device to display the virtual screen in the second display area.

2. The method according to claim 1, characterized in that The method further comprises: When detecting that the first terminal device is adjusted from the first display state to the second display state, adjusting the content display mode of the first main screen according to the second display state to obtain the second main screen, and reorganizing the first display area and the second display area according to the second display state; The second main screen and the virtual screen are projected to the second terminal device, and the third position information of the reorganized first display area and the fourth position information of the reorganized second display area are sent to the second terminal device, the third position information is used by the second terminal device to display the second main screen in the reorganized first display area, and the fourth position information is used by the second terminal device to display the virtual screen in the reorganized second display area.

3. The method according to claim 1 or 2, characterized in that: Before projecting the virtual screen to the second terminal device, the method includes: When obtaining the application startup instruction sent by the second terminal device, the virtual screen is created according to the first main screen, and the application interface of the application is displayed on the virtual screen, and the virtual screen is not displayed in the first terminal device.

4. The method according to any one of claims 1 to 3, characterized in that The determining, according to the first display state of the first terminal device, a first display area of ​​the first main screen of the first terminal device in the second terminal device and a second display area of ​​the virtual screen of the first terminal device in the second terminal device includes: Acquire interface information of the second terminal device; The first display area and the second display area are determined according to a first display state of the first terminal device and interface information of the second terminal device.

5. The method according to claim 4, characterized in that The interface information includes an interface height and an interface width, and the determining the first display area and the second display area according to the first display state of the first terminal device and the interface information of the second terminal device includes: When the first display state is a vertical screen display state, it is determined that a first area height of the first display area is the same as a second area height of the second display area, and the first area height and the second area height are the same as the interface height.

6. The method according to claim 5, characterized in that The method further comprises: It is determined that a first area width of the first display area is the same as a screen width of the first main screen.

7. The method according to claim 4, characterized in that The interface information includes an interface height and an interface width, and the determining the first display area and the second display area according to the first display state of the first terminal device and the interface information of the second terminal device includes: When the first display state is a horizontal screen display state, it is determined that a first area width of the first display area is the same as a second area width of the second display area, and the first area width and the second area width are the same as the interface width.

8. The method according to claim 7, characterized in that The method further comprises: It is determined that a first area height of the first display area is the same as a screen height of the first main screen.

9. The method according to claim 4, characterized in that The first terminal device is a terminal device with a foldable display screen, and the interface information includes an interface height and an interface width; The determining the first display area and the second display area according to the first display state of the first terminal device and the interface information of the second terminal device includes: When the first display state is an expanded screen display state, determining a third display state of the second terminal device according to the interface height and the interface width; The first display area and the second display area are determined according to the third display state and the interface information.

10. The method according to claim 9, characterized in that The determining the first display area and the second display area according to the third display state and the interface information includes: When the third display state is the horizontal screen display state, it is determined that the first area height of the first display area is the same as the second area height of the second display area, and the first area height, the second area height and the interface height are the same.

11. The method according to claim 10, characterized in that The method further comprises: It is determined that a first area width of the first display area is the same as a screen width of the first main screen, or is the same as a screen height of the first main screen.

12. The method according to claim 9, characterized in that The determining the first display area and the second display area according to the third display state and the interface information includes: When the third display state is a vertical screen display state, it is determined that a first area width of the first display area is the same as a second area width of the second display area, and the first area width and the second area width are the same as the interface width.

13. The method according to claim 12, characterized in that The method further comprises: It is determined that a first area height of the first display area is the same as a screen height of the first main screen, or is the same as a screen width of the first main screen.

14. A collaborative display method, applied to a second terminal device, characterized in that: The method comprises: Acquire a first display state of the first terminal device, and determine a first display area of ​​a first main screen of the first terminal device in the second terminal device and a second display area of ​​a virtual screen of the first terminal device in the second terminal device according to the first display state; Obtaining a first main screen projected by the first terminal device, and displaying the first main screen in the first display area; Obtain the virtual screen projected by the first terminal device, and display the virtual screen in the second display area.

15. The method according to claim 14, characterized in that The method further comprises: When it is determined that the first terminal device is adjusted from the first display state to the second display state, reorganizing the first display area and the second display area according to the second display state; Obtain the second main screen and the virtual screen projected by the first terminal device, and display the second main screen in the reorganized first display area, and display the virtual screen in the reorganized second display area. The content display method of the second main screen corresponds to the second display state.

16. The method according to claim 14 or 15, characterized in that Before obtaining the virtual screen projected by the first terminal device, the method includes: In response to a start-up operation on an application in the first main screen, a start-up instruction for starting the application is sent to the first terminal device, where the start-up instruction is used by the first terminal device to start the application and display the application interface of the application on the virtual screen.

17. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the terminal device implements the method according to any one of claims 1 to 16.

18. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a computer, the computer is caused to implement the method according to any one of claims 1 to 16.

Images