Screen projection control method and electronic device

Abstract

A screen projection control method and an electronic device. A first electronic device displays a first interface of a first application; in response to a screen projection operation, the first electronic device sends display information of the first interface to a second electronic device; the second electronic device displays the first interface of the first application; the first electronic device displays a second interface of a second application and a first window, the first window including a third interface, the third interface being the same as a current display interface of the first application projected to the second electronic device; the first electronic device updates the third interface in the first window to a fourth interface in response to a user operation; and the second electronic device updates the display interface of the first application to the fourth interface. In this way, the sender in the screen projection scenario can conveniently control the display information projected to the receiver.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and in particular to a screen projection control method and electronic device. Background Technology

[0002] Screen mirroring is a rapidly emerging technology that allows electronic devices to project their displays onto other electronic devices. For example, a movie on a mobile phone can be projected onto a television for playback, or pictures on a tablet can be projected onto a television for display.

[0003] Taking screen mirroring from a mobile phone to a TV as an example, generally speaking, after mirroring, users can perform other operations on their phones; this type of mirroring is called app mirroring. For instance, if a mobile phone mirrors a TV series from a video player to the TV, the user can chat using an instant messaging app on their phone. When the user needs to control the content mirrored to the TV (such as playing the next episode), the phone needs to switch from app mirroring to screen mirroring mode (i.e., the phone and TV display the same interface) to perform these actions. This method, which requires constant switching between app mirroring and screen mirroring, is cumbersome and provides a poor user experience. Summary of the Invention

[0004] The purpose of this application is to provide a screen projection control method and electronic device to improve the convenience of controlling the target device in screen projection scenarios.

[0005] Firstly, a screen mirroring control method is provided. This method can be applied to a system including a first electronic device and a second electronic device. The method includes: the first electronic device displaying a first interface of a first application; in response to a screen mirroring operation, the first electronic device sending display information of the first interface to the second electronic device; the second electronic device displaying the first interface of the first application; the first electronic device displaying a second interface of a second application and a first window, the first window including a third interface, the third interface being the same as the currently displayed interface of the first application mirrored to the second electronic device; in response to a user operation, the first electronic device updating the third interface in the first window to a fourth interface; the first electronic device sending display information of the fourth interface to the second electronic device; and the display interface of the first application on the second electronic device being updated to the fourth interface.

[0006] In other words, when the first electronic device projects its screen onto the second electronic device (such as through app projection), a first window is displayed on the screen of the first electronic device. The information displayed in the first window is the same as the information projected onto the second electronic device. This allows the user to control the information projected onto the second electronic device through the first window, improving the convenience of controlling the projected information on the target device (i.e., the second electronic device) in projection scenarios.

[0007] In one possible design, the first window floats above the second interface, and the first window does not completely cover the second interface; or, the second interface and the first window are displayed in a split-screen manner. That is, after the first electronic device launches the first application, the first electronic device displays the second interface of the second application and the first window. There can be various methods for displaying the first window and the second interface, such as split-screen display, or the first window being located on top, etc., which are not limited in the embodiments of this application.

[0008] In one possible design, the first electronic device displays a first window, including: receiving an operation to invoke the first window; and displaying the first window in response to the operation. That is, after the first electronic device projects a first application, it displays a second interface of a second application, at which point the first window may not be displayed. The first window is invoked when the user triggers the operation. This allows the user to invoke the first window when needed (e.g., when they need to control the information displayed on the target device), resulting in a better user experience.

[0009] In one possible design, displaying the first window in response to the operation of activating the first window includes: displaying a first marker on the second interface; and activating the first window in response to an operation on the first marker. For example, the first marker can be any form, such as a button, icon, or text. That is, after the first electronic device launches the first application, it displays the second interface of the second application and the first marker, which is used to activate the first window. The display area of ​​the first marker is smaller than the display area of ​​the first window, so it does not obstruct the second interface excessively, and the display of the first marker can provide some operational guidance for the user to activate the first window, resulting in a better user experience.

[0010] In one possible design, the method further includes: hiding the first window in response to an operation to hide the first window; or adjusting the display position and / or display area of ​​the first window in response to an operation to adjust the display position and / or display area of ​​the first window. That is, after the first electronic device launches the first application, the first electronic device displays the second interface of the second application and the first window. If the user does not want to display the first window (e.g., the first window obscures the second interface), they can hide the first window or adjust its area, position, etc., resulting in a better user experience.

[0011] In one possible design, the first electronic device displays a first window, comprising: the first electronic device creating a drawing surface, wherein display information on the drawing surface is displayed via a display screen of the first electronic device; the first electronic device mapping display information in a first virtual display screen (VD) onto the drawing surface; wherein the first application runs in the first VD, and the first VD is used to send display information within the first VD to a second electronic device.

[0012] In other words, when the first electronic device wants to project the first application, it transfers the first application to the first VD (Visual Display Device) for execution, so that the second electronic device can display the first interface of the first application. The display information of the first VD can also be mapped to a drawing surface (i.e., the first window) corresponding to the local display screen, so that the information projected onto the second electronic device is displayed in the first window on the local display screen. Simply put, the first application running in the first VD is sent to both the second electronic device and the local display screen (i.e., the first window) for display, achieving synchronization of the projected information between the local display screen and the second electronic device, allowing the user to control the displayed information on the second electronic device through the local display screen (i.e., the first window).

[0013] In one possible design, the first electronic device maps display information in a first virtual display screen (VD) onto the drawing surface, including: the first electronic device creating a second VD corresponding to the drawing surface; the first electronic device mapping display information in the first VD to the second VD; and the display information in the second VD being displayed through the drawing surface.

[0014] It's important to note that, generally, the first electronic device has a built-in mapping function (such as mirror mapping), meaning the mirror mapping function can be used directly without modifying the underlying code. Mapping from the first VD to the second VD can use the system's built-in mirror mapping, thus avoiding modifications to the underlying code. Furthermore, the mapping from the second VD to the phone's local display is a surface mapping, which also doesn't require underlying processing. Therefore, this method is relatively easy to implement and has high processing efficiency.

[0015] In one possible design, in response to a user operation, the first electronic device updates the third interface within the first window to a fourth interface, including: in response to a user operation, updating the information displayed on the drawing interface to the fourth interface; wherein, the display information of the fourth interface on the drawing interface is mapped to a second VD, the second VD is used to map the display information of the fourth interface to the first VD, and the first VD is used to send the display information of the fourth interface to the second electronic device. In this way, the first electronic device can control the screen projection information on the second electronic device through the first window without modifying the underlying code, resulting in low implementation difficulty and high efficiency.

[0016] In one possible design, the method further includes: the first electronic device sending the display information of the fourth interface to the third electronic device; the third electronic device displaying the fourth interface; the first electronic device transferring control of the display information projected onto the second electronic device to the third electronic device; and when the third electronic device controls the fourth interface on its own display screen to be updated to the fifth interface, the second electronic device updating the fourth interface on its own display screen to the fifth interface.

[0017] In other words, the first electronic device can project its screen onto the second and third electronic devices, and transfer control of the second electronic device to the third electronic device, relieving the processing pressure on the first electronic device, allowing the first electronic device to perform other operations without being disturbed.

[0018] In one possible design, the method further includes: de-displaying the first window on the first electronic device. Therefore, after the first electronic device transfers control of the second electronic device to the third electronic device, the first window may not be displayed on the first electronic device, thus avoiding obstruction of the display screen.

[0019] In one possible design, the first electronic device transfers control of the display information projected onto the second electronic device to a third electronic device, including: the first electronic device creating a third VD; the third VD being used to send the display information within the third VD to the third electronic device; and the first electronic device mapping the display information in the first VD onto the third VD; wherein the first application runs in the first VD, and the first VD is used to send the display information within the first VD to the second electronic device. In this embodiment, the transfer of control is achieved through information mapping between multiple VDs, without requiring modification to the underlying code. This method is relatively easy to implement and has high processing efficiency.

[0020] In one possible design, when the third electronic device controls the fourth interface on its local display screen to update to the fifth interface, the second electronic device updates the fourth interface on its local display screen to the fifth interface. This includes: when the third electronic device updates the fourth interface to the fifth interface, the display information in the third VD is updated to the display information of the fifth interface. The third VD is used to map the display information of the fifth interface to the first VD, and the first VD is used to send the display information of the fifth interface to the second electronic device. Therefore, after receiving control of the screen projection information (the information projected from the first electronic device to the second electronic device) on the second electronic device, the third electronic device controls the screen projection information of the second electronic device through information mapping between multiple VDs. This process does not require modification of the underlying code, has low implementation difficulty, and high processing efficiency.

[0021] Secondly, a screen casting control method is provided, comprising: a first electronic device displaying a first interface of a first application; in response to a screen casting operation, the first electronic device sending the display information of the first interface to a second electronic device;

[0022] The first electronic device displays a second interface of the second application and a first window, the first window including a third interface, the third interface being the same as the current display interface of the first application projected onto the second electronic device;

[0023] The first electronic device responds to the user's operation by updating the third interface in the first window to a fourth interface;

[0024] The first electronic device sends the display information of the fourth interface to the second electronic device, so that the display interface of the first application on the second electronic device is updated to the fourth interface.

[0025] In one possible design, the first window is displayed floating above the second interface, and the first window does not completely cover the second interface; or, the second interface and the first window are displayed in a split-screen format.

[0026] In one possible design, the first electronic device displays a first window, including:

[0027] Received an operation to invoke the first window;

[0028] In response to the operation of bringing up the first window, the first window is displayed.

[0029] In one possible design, displaying the first window in response to the operation of invoking the first window includes: displaying a first marker on the second interface;

[0030] In response to an operation on the first mark, the first window is invoked.

[0031] In one possible design, the method further includes:

[0032] In response to the operation of hiding the first window, the first window is hidden; or,

[0033] In response to an operation for adjusting the display position and / or display area of ​​the first window, the display position and / or display area of ​​the first window are adjusted.

[0034] In one possible design, the first electronic device displays a first window, including:

[0035] The first electronic device creates a drawing surface, and the display information on the drawing surface is displayed through the display screen of the first electronic device;

[0036] The first electronic device maps the display information in the first virtual display screen VD onto the drawing surface;

[0037] The first application runs in the first VD, and the first VD is used to send display information within the first VD to the second electronic device.

[0038] In one possible design, the first electronic device maps display information from the first virtual display screen VD onto the drawing surface, including:

[0039] The first electronic device creates a second VD, which corresponds to the drawing surface;

[0040] The first electronic device maps the display information in the first VD to the second VD, and the display information in the second VD is displayed through the drawing surface.

[0041] In one possible design, in response to a user action, the first electronic device updates the third interface within the first window to a fourth interface, including:

[0042] In response to a user operation, the information displayed on the drawing interface is updated to the fourth interface; wherein, the display information of the fourth interface on the drawing interface is mapped to the second VD, the second VD is used to map the display information of the fourth interface to the first VD, and the first VD is used to send the display information of the fourth interface to the second electronic device.

[0043] In one possible design, the method further includes:

[0044] The first electronic device sends the display information of the fourth interface to the third electronic device, so that the third electronic device displays the fourth interface;

[0045] The first electronic device transfers control of the display information projected onto the second electronic device to the third electronic device.

[0046] In one possible design, the method further includes: de-displaying the first window on the first electronic device.

[0047] In one possible design, the first electronic device transfers control of the display information projected onto the second electronic device to a third electronic device, including:

[0048] The first electronic device creates a third VD; the third VD is used to send display information within the third VD to the third electronic device;

[0049] The first electronic device maps display information in the first VD to the third VD; wherein the first application runs in the first VD, and the first VD is used to send the display information in the first VD to the second electronic device.

[0050] In one possible design, the method further includes: the third VD is used to update the display information in the third VD to the display information of the fifth interface when the fourth interface on the third electronic device is updated to the fifth interface, and to map the display information of the fifth interface to the first VD, wherein the first VD is used to send the display information of the fifth interface to the second electronic device.

[0051] Thirdly, a communication system is also provided, comprising: a first electronic device and a second electronic device;

[0052] The first electronic device includes: a processor; a memory; wherein the memory stores one or more computer programs, the one or more computer programs including instructions that, when executed by the processor, cause the first electronic device to perform the steps of the first electronic device as described in the first aspect above.

[0053] The second electronic device includes: a processor; a memory; wherein the memory stores one or more computer programs, the one or more computer programs including instructions that, when executed by the processor, cause the second electronic device to perform the steps of the second electronic device as described in the second aspect above.

[0054] Fourthly, an electronic device is also provided, comprising:

[0055] Processor, memory, and one or more programs;

[0056] The one or more programs are stored in the memory, and the one or more programs include instructions that, when executed by the processor, cause the electronic device to perform the method steps provided in the second aspect above.

[0057] Fifthly, a computer-readable storage medium is also provided for storing a computer program that, when run on a computer, causes the computer to perform the methods provided in the first or second aspect above.

[0058] In a sixth aspect, a computer program product is also provided, comprising a computer program that, when run on a computer, causes the computer to perform the method described in the first or second aspect above.

[0059] A seventh aspect also provides a graphical user interface for an electronic device, the electronic device having a display screen, a memory, and a processor, the processor being configured to execute one or more computer programs stored in the memory, the graphical user interface including a graphical user interface displayed when the electronic device performs the method described in the second aspect above.

[0060] Eighthly, this application also provides a chip coupled to a memory in an electronic device, used to call a computer program stored in the memory and execute the technical solutions of the first to second aspects of this application. In this application, "coupling" means that two components are directly or indirectly combined with each other.

[0061] The beneficial effects of aspects two through eight mentioned above are the same as those of aspect one, and will not be repeated here. Attached Figure Description

[0062] Figures 1A to 1B This is a schematic diagram illustrating screen mirroring according to an embodiment of this application;

[0063] Figures 2A to 2B A schematic diagram illustrating application screen mirroring according to an embodiment of this application;

[0064] Figure 3 This is a schematic diagram illustrating the switching between application casting and mirror casting according to an embodiment of this application;

[0065] Figures 4A to 4B A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0066] Figures 5A to 5B A schematic diagram illustrating another application scenario provided by an embodiment of this application;

[0067] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0068] Figure 7 A schematic diagram illustrating the application screen mirroring on the main device provided in this embodiment of the application;

[0069] Figure 8 A schematic diagram of the background processing flow for screen mirroring of the main device application provided in the embodiments of this application;

[0070] Figure 9 A first schematic diagram of a main device displaying a first window provided in an embodiment of this application;

[0071] Figures 10 to 11 A second schematic diagram illustrating the main device displaying the first window provided in this application embodiment;

[0072] Figures 12 to 13 A schematic diagram illustrating how a master device controls a target device through a first window, as provided in this application embodiment;

[0073] Figure 14 A schematic flowchart of the screen projection control method provided in the embodiments of this application;

[0074] Figure 15 Another flowchart illustrating the screen projection control method provided in this application embodiment;

[0075] Figure 16 A schematic diagram of the software structure of the main device provided in the embodiments of this application;

[0076] Figure 17This is another flowchart illustrating the screen projection control method provided in the embodiments of this application;

[0077] Figures 18 to 19 A schematic diagram illustrating the projection of a main device onto multiple target devices, provided in an embodiment of this application.

[0078] Figure 20 A flowchart illustrating the screen projection control method provided in this application embodiment;

[0079] Figure 21 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0080] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0081] (1) An application (app) is a software program that performs one or more specific functions. Typically, multiple applications can be installed on an electronic device, such as instant messaging applications, video applications, audio applications, image capture applications, etc. Instant messaging applications, for example, may include SMS applications, MeeTime, etc. WhatsApp Photo sharing (Instagram), Kakao Applications that capture images, such as camera apps (system camera or third-party camera apps). Applications that capture videos, such as... And so on. Audio applications, such as Google Music, Kugou Music, Xiami, QQ Music, etc. The applications mentioned in the following examples can be those pre-installed on the electronic device at the factory, or those downloaded by the user from the network or obtained from other electronic devices during use.

[0082] (2) The at least one involved in the embodiments of this application includes one or more; wherein, more than two means two or more. In addition, it should be understood that in the description of this application, the words "first", "second" and the like are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance, nor should they be construed as indicating or implying order.

[0083] The technical solutions provided in this application are applicable to screen mirroring scenarios. Generally, a screen mirroring scenario includes a sending end and a receiving end. The sending end sends display information to the receiving end for display. For ease of description, the sending end is referred to as the main device and the receiving end as the target device. The main device can be a mobile phone, tablet, PC, watch, etc. The target device can be a large-screen device such as a tablet, PC, or television. It is understood that, besides main device and target device, the sending end and receiving end can have other names, such as the sending end being an active screen mirroring device and the receiving end being a passive screen mirroring device; or the sending end being the main device and the receiving end being an auxiliary device; or the sending end being the source device and the receiving end being the target device; or the sending end being the first electronic device and the receiving end being the second electronic device, etc. This application does not limit these specific names. The roles of the sending end and the receiving end can be interchanged; that is, the sending end can mirror to the receiving end, and correspondingly, the receiving end can also mirror to the sending end.

[0084] Screen mirroring technology includes screen mirroring and app mirroring.

[0085] Screen mirroring refers to the process where the encoder at the sending end encodes the currently displayed information on the sending end's screen, sends the encoded information to the receiving end via a data channel, and the receiving end decodes the received information using a decoder, creating a Surface to display the decoded content. In this way, the sending and receiving ends can display the same image; in other words, whatever content is currently displayed at the sending end is what is projected onto the receiving end. For an example, please refer to [link to example]. Figure 1A The diagram illustrates the screen mirroring process. The sending end (e.g., a mobile phone) mirrors the foreground display information (e.g., interface A of the first application) to the receiving end (e.g., a PC), and the receiving end displays the displayed information (e.g., interface A of the first application). When the foreground display information on the sending end changes, for example, switching from interface A to interface B, the corresponding display information on the receiving end also changes, i.e., switching from interface A to interface B.

[0086] One way to implement screen mirroring is to directly encode the current display information of the sending end's local display and send it to the sending end, where it is decoded and displayed. This method does not require creating a virtual display (VD). Another way to implement screen mirroring is by creating a VD; for example, please refer to [link to relevant documentation]. Figure 1BAs shown, the phone creates a first Virtual Display (VD) and maps the current display information from the phone's local screen to the first VD. The first VD is invisible to the user; it's a virtual display created in the phone's background, and the display information in the first VD is projected onto the target device by default. For example, the first VD sends the display information to an encoder for encoding, and the encoded display information is sent to a receiving end (such as a PC or TV). In this way, the phone projects its foreground display information onto the receiving end. See also... Figure 1B In this unidirectional data flow method, the data source is the phone's local display screen. The information displayed on the local display screen is mapped to the first VD in real time. The display that needs to be encoded is the first VD, and the encoded data is sent to the receiving end. Therefore, when the information displayed on the phone's local display screen changes, the corresponding information displayed on the receiving end also changes accordingly.

[0087] Application casting differs from screen mirroring. Simply put, in application casting scenarios, the information displayed on the receiving end, cast from the sending end, can differ from the information currently displayed on the sending end. For an example, please refer to [link to example]. Figure 2A The diagram illustrates the application casting process. The main device displays the interface A of the first application. If the main device detects an application casting operation, it will cast the first application to the target device, while the main device displays the interface of the second application in the foreground. Essentially, the main device casts the entire first application to the target device; in short, the casting granularity is the application level.

[0088] For the implementation principle of application screen mirroring, please refer to [link / reference]. Figure 2B .like Figure 2B In (a) of the example, the phone's local display is showing the interface A of the first application. If the phone detects an application casting operation, it will transfer the first application to run on the first VD (e.g., move the first application from task stack 1 on the local display to task stack 2 on the first VD). Figure 2B In (b) of the diagram, after the first application is moved to the first VD, the interface of the second application is displayed on the local screen. The first VD sends the display information to the encoder for encoding, and the encoded display information is then sent to the receiving end. Because the first application is moved to the first VD, when the display interface of the first application in the first VD changes, the corresponding display interface of the first application on the receiving end also changes. The display interface on the local screen can be different from the display interface projected onto the first VD and the receiving end. Simply put, the data source for projecting to the target device is no longer the phone's local screen, but the first VD.

[0089] In some embodiments, screen mirroring and application casting can be switched. For example, if the current main device and target device are using screen mirroring, a switching operation can be performed to switch from screen mirroring to application casting. Alternatively, if the current main device and target device are using application casting, a switching operation can also be performed to switch from application casting to screen mirroring.

[0090] The following example demonstrates how to switch from application casting to mirror casting.

[0091] Please see Figure 3 The image shows a schematic diagram of switching from application casting to screen mirroring. Specifically, the switching process includes: (See...) Figure 3 In step (a), the first application in the first VD (i.e., the application that was projected) is rolled back to the local display screen. The rollback operation can be understood as a stack shifting operation (for example, moving the first application from task stack 2 in the first VD to task stack 1 in the local display screen). After the first application is rolled back from the first VD to the local display screen, the first application in the first VD is released, that is, there is no first application in the first VD. At this time, the application projection ends.

[0092] Please see Figure 3 In (b) of the example, after the first application returns to the local display screen, the local display screen uses screen mirroring for projection. That is, the local display screen maps the first application to the first VD (Virtual Display Device), and the first VD sends the display information to the target device. In other words, Figure 3 (b) in the middle and the previous one Figure 1B The mirroring process is the same. At this point, the switch from application casting to mirror casting is complete.

[0093] It's important to note that in app casting, since the cast app (the primary app) isn't displayed on the phone's local screen, users need to see its interface on the local screen to control it. One approach is to switch from app casting to mirroring, allowing the user to control the primary app on their local screen, thus controlling the cast app on the target device. For example, see... Figure 3 In (c), the user controls the first application on the local display screen, causing the interface A of the first application to switch to interface B, which is synchronized to the first VD. Then the first VD is synchronized to the target device, and the interface A of the first application is updated to interface B of the first application on the target device.

[0094] This method of controlling the screen mirroring of applications on the target device requires switching from application mirroring to screen mirroring, which is cumbersome, time-consuming, and results in a poor user experience.

[0095] This application provides a method for controlling the information displayed on a target device when the main device casts an application to the target device. When the main device casts an application to the target device, a first window can be displayed on the main device. The first window displays the current display interface of the application cast to the target device. That is, the display information in the first window is synchronized with the current display information of the application cast to the target device. The user can control the display information cast to the target device within the first window.

[0096] Figures 4A to 4B This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. For example... Figure 4A In (a), the phone is currently displaying the home screen of the Photos app. When the phone detects an app casting operation, it casts the Photos app to the target device, where the Photos app's interface is displayed, such as... Figure 4A (b) in the example. Figure 4A In (c), after the photo album application is projected, the phone displays the instant messaging application's interface (such as a chat interface with a contact), while the target device still displays the photo album application's interface, such as... Figure 4A (d) in the middle.

[0097] In some embodiments, in display Figure 4A During the interface process shown in (c), if the mobile phone detects an operation used to control the display information cast to the target device (e.g., detecting a user click), Figure 4A When the first window 401 is displayed on the phone (as indicated by marker 400 in interface (c) or when a voice command is received to instruct the user to control the display information cast to the target device), the phone displays the first window 401. Figure 4B In (a) of this method, the user can bring up the first window 401 when needed by triggering marker 400 or by using a voice command. In other embodiments, the first window 401 may appear automatically after the phone projects the photo album application. For example, Figure 4A In (a) of the example, after the phone casts the photo album application to the target device, the phone does not display it. Figure 4A The interface shown in (c) directly displays... Figure 4B The interface shown in (a) is shown in the image.

[0098] The first window 401 can be displayed in a floating manner, completely or partially obscuring the interface of the instant messaging application. The first window 401 displays the current display interface (i.e., the photo album application homepage) of the photo album application projected onto the target device. In other words, the first window 401 is identical to the current display interface of the projected application on the target device. This allows the user to control the photo album application projected onto the target device through the first window 401. For example, please refer to... Figure 4BIn (c), if the user opens an image in the photo album application in the first window 401, then the corresponding image opened in the photo album application on the target device, such as... Figure 4B (d) In this context, users can also perform other operations on the photo album application through the first window 401, such as swiping to the next image, zooming in and out of images, opening the photo album homepage, etc.

[0099] Figures 5A to 5B This is a schematic diagram illustrating another application scenario provided by an embodiment of this application. For example... Figure 5A In (a), the phone is displaying the interface of a video playback application (e.g., the playback interface of a TV series). When the phone detects an application casting operation, it casts the video playback application to the target device, such as... Figure 5A In (b) of the diagram, the video playback application interface is displayed on the target device. After casting the video playback application, the main page is displayed on the phone's foreground, as shown below. Figure 5A (c) in the middle, while the target device still displays the interface of the video playback application, such as Figure 5A (d) in the middle.

[0100] In some embodiments, in display Figure 5A During the interface process shown in (c), if the mobile phone detects an operation used to control the display information cast to the target device (e.g., detecting a user click), Figure 5A In interface (c), if marker 500 is received, or if a voice command is received instructing the user to control the display information cast to the target device, the first window 501 is displayed on the phone, such as... Figure 5B (a) In other embodiments, the first window 501 may appear automatically after the mobile phone launches the video playback application. For example, Figure 5A In (a) of the example, after the mobile phone casts the video playback application to the target device, the mobile phone does not display the video. Figure 5A The interface shown in (c) directly displays... Figure 5B The interface shown in (a) is shown in the image.

[0101] The first window 501 can be displayed in a floating manner, and can completely or partially obscure the main interface. The first window 501 displays the current display interface of the video playback application cast to the target device (i.e., the playback interface of the TV series). In other words, the first window 501 is identical to the current display interface of the casting application on the target device. In this way, the user can control the video playback application cast to the target device through the first window 501. (See also...) Figure 5B In (c), the user opens the next episode of the TV series in the first window 501. Correspondingly, the next episode of the TV series is opened on the target device, such as... Figure 5B(d) In this context, users can also perform other operations on the video playback application through the first window 501, such as speeding up playback, slowing down playback, selecting other TVs, etc.

[0102] The above examples illustrate two application scenarios of the embodiments of this application. It should be noted that this application is also applicable to other application scenarios.

[0103] The main device is described below. The main device can be a portable electronic device, such as a mobile phone, tablet computer, laptop computer, wearable device with wireless communication capabilities (such as a smartwatch, smart glasses, smart bracelet, or smart helmet), or in-vehicle device. Exemplary embodiments of the portable electronic device include, but are not limited to, carrying... Alternatively, it can be a portable electronic device with another operating system. The target device can be a tablet, PC, television, or other large-screen device.

[0104] Figure 6 A schematic diagram of the electronic device is shown. Figure 6 The structure can be the structure of the main equipment or the structure of the target equipment. For example... Figure 6 As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0105] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, memory, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). Different processing units may be independent devices or integrated into one or more processors. The controller may serve as the nerve center and command center of the electronic device. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. Processor 110 may also include memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that processor 110 has just used or is recurring. If processor 110 needs to reuse an instruction or data, it can directly retrieve it from the memory. This avoids repeated access, reduces the waiting time of processor 110, and thus improves system efficiency.

[0106] USB interface 130 is a USB standard compliant interface, specifically a Mini USB interface, Micro USB interface, USB Type-C interface, etc. USB interface 130 can be used to connect a charger to charge electronic devices, and can also be used for data transfer between electronic devices and peripheral devices. Charging management module 140 receives charging input from the charger. Power management module 141 connects to battery 142, charging management module 140, and processor 110. Power management module 141 receives input from battery 142 and / or charging management module 140, providing power to processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.

[0107] The wireless communication function of the electronic device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in conjunction with a tuning switch.

[0108] The mobile communication module 150 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G in electronic devices. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0109] The wireless communication module 160 can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0110] In some embodiments, antenna 1 of the electronic device is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling the electronic device to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0111] Display screen 194 is used to display the application's display interface, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include one or N displays 194, where N is a positive integer greater than 1.

[0112] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0113] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0114] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1.

[0115] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system and software code for at least one application (e.g., iQiyi application, WeChat application, etc.). The data storage area may store data generated during the use of the electronic device (e.g., images, videos, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0116] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, images, videos, and other files can be saved on the external memory card.

[0117] Electronic devices can implement audio functions such as music playback and recording through audio modules 170, speakers 170A, receivers 170B, microphones 170C, headphone jacks 170D, and application processors.

[0118] Pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, pressure sensor 180A may be disposed on display screen 194. Gyroscope sensor 180B can be used to determine the motion posture of electronic device. In some embodiments, the angular velocity of electronic device about three axes (i.e., x, y, and z axes) can be determined by gyroscope sensor 180B.

[0119] The gyroscope sensor 180B can be used for image stabilization. The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device calculates altitude using the air pressure value measured by the barometric pressure sensor 180C, assisting in positioning and navigation. The magnetic sensor 180D includes a Hall effect sensor. The electronic device can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In some embodiments, when the electronic device is a flip phone, it can detect the opening and closing of the flip cover using the magnetic sensor 180D. Based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set. The accelerometer sensor 180E can detect the magnitude of the acceleration of the electronic device in various directions (generally three axes). When the electronic device is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the electronic device, applicable to applications such as screen orientation switching and pedometers.

[0120] A proximity sensor 180F is used to measure distance. The electronic device can measure distance using infrared or laser. In some embodiments, during a shooting scene, the electronic device can utilize the proximity sensor 180F for distance measurement to achieve fast focusing. A proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device emits infrared light outward through the LED. The electronic device uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that an object is near the electronic device. When insufficient reflected light is detected, the electronic device can determine that no object is near the electronic device. The electronic device can use the proximity sensor 180G to detect when a user holds the electronic device close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and screen locking.

[0121] The ambient light sensor 180L is used to sense ambient light intensity. Electronic devices can adaptively adjust the brightness of the display 194 based on the sensed ambient light. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking photos. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the electronic device is in a pocket, preventing accidental touches. The fingerprint sensor 180H is used to collect fingerprints. Electronic devices can use the collected fingerprint characteristics to achieve fingerprint unlocking, app access lock, fingerprint photography, fingerprint answering of calls, etc.

[0122] Temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device uses the temperature detected by temperature sensor 180J to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, the electronic device reduces the performance of a processor located near temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is below another threshold, the electronic device heats battery 142 to prevent abnormal shutdown of the electronic device due to low temperature. In still other embodiments, when the temperature is below yet another threshold, the electronic device boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0123] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of the electronic device, in a different position than display screen 194.

[0124] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from vibrating bone fragments in the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals.

[0125] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. The electronic device can receive button inputs and generate key signal inputs related to user settings and function control. Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations applied to different applications (such as taking photos, audio playback, etc.). Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, or messages, missed calls, notifications, etc. SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the electronic device.

[0126] Understandable Figure 6 The components shown do not constitute a specific limitation on the electronic device. The electronic device in the embodiments of the present invention may include, but is not limited to, components that are more advanced than those shown. Figure 6 More or fewer parts. Furthermore, Figure 6 The combination / connection relationships between the components can also be adjusted and modified.

[0127] The technical solutions provided by the embodiments of this application are described below with reference to the accompanying drawings.

[0128] For ease of description, the following embodiments are illustrated using a mobile phone as the main device and a television as the target device.

[0129] Example 1

[0130] Figure 7Image (a) shows a graphical user interface (GUI) of a mobile phone, which serves as the phone's desktop. The desktop includes icons for different applications. When the phone detects an action on the icon of the first application, it displays... Figure 7 The interface of the first application is shown in (b) above. The first application can be any application, such as a video playback application, browser, instant messaging application, photo album application, etc. When the phone detects a screen mirroring operation, it displays a list of target devices, such as... Figure 7 In section (c), the target device list includes identifiers for target devices such as televisions and PCs, allowing users to select the target device for screen mirroring. Screen mirroring operations can take various forms. For example, a screen mirroring button may be displayed in the swipe-down notification bar (accessed by swiping down from the top of the screen) or the pull-up control center (accessed by swiping up from the bottom of the screen), and the screen mirroring operation refers to clicking this button. Alternatively, screen mirroring operations can also be voice commands indicating screen mirroring. This application embodiment does not limit the type of screen mirroring operation. Figure 7 In section (c), the target device list includes identifiers for target devices such as televisions and PCs. Assuming the phone detects the user selecting the television identifier from the target device list, in response to this action, the foreground application (i.e., the first application) is cast to the television screen, such as... Figure 7 (d) shows the interface of the first application displayed on the television.

[0131] After casting the first app from your phone to the TV, the interface of the second app will be displayed in the foreground on your phone, such as... Figure 7 (e) In this context, the second application can be the application that was previously opened before the first application, or it can be the phone's home screen, or it can be the interface of a pre-configured application (either the system default setting or the user setting).

[0132] The above describes the app casting process. After the phone casts the first app, the interface of the second app is displayed in the foreground. For the corresponding background processing flow of the phone, please refer to [link to relevant documentation]. Figure 8 .like Figure 8 The phone creates a first Visual Display (VD) (optionally, the display parameters of the first VD are adapted to the target device, such as the same resolution). The phone then transfers the first application from its local display to the first VD to run. The first VD then projects the first application onto the target device. After the first application is transferred from the local display to the first VD, the interface of the second application is displayed on the local display. It should be noted that... Figure 8 The fact that the interface of the first application and the interface of the second application are smaller than the area of ​​the local display screen is to indicate that the local display screen can display the interface of the first application or the interface of the second application, and does not limit the size relationship between the application interface and the local display screen.

[0133] When casting to an app, users can perform other operations on their phones, and they can also control the first app cast to the target device from their phones.

[0134] The first method

[0135] When a phone casts the first application to a target device, the phone displays the interface of the second application and also shows the first window. The first window displays the current display interface of the first application on the target device; in other words, the interface displayed in the first window is the same as the current display interface of the first application on the target device. Users can control the first application cast to the target device through the first window.

[0136] For example, please see Figure 9 In (a), when the phone detects an action to select a TV icon from the target device list, it casts the first application to the TV. For example... Figure 9 In (b) of the example, the interface of the first application is displayed on the television screen. For example... Figure 9 In step (c), after the first application is projected, the phone displays the interface of the second application, as well as the first window 901. The display interface in the first window 901 is the same as the current display interface of the first application projected onto the target device.

[0137] In other words, after the phone uses app casting to project the first app, the phone screen displays the interface of the second app, and the first window 901 automatically appears, so that the user can control the first app projected onto the target device through the first window 901.

[0138] The first window 901 can have multiple display modes, such as... Figure 9 In (c), the first window 901 is displayed floating on the interface of the second application, and can completely or partially cover the interface of the second application. The shape of the first window 901 is not limited in this embodiment.

[0139] The display position and / or display area of ​​the first window 901 can be preset by default, or adjustable by the user. For example, the first window 901 may include zoom buttons 902, which are used to zoom in or out of the first window. When zooming in, the first window can be displayed in full screen; when zooming out, the first window can be reduced to occupy the minimum area (e.g., the minimum area set by the system).

[0140] The first window 901 can also be hidden. For example, when the phone detects a hide command to hide the first window, the first window 901 is hidden. See also [example description]. Figure 9In (c), the first window 901 includes a close button 903. When an operation on the close button 903 is detected, the first window 901 is hidden. Alternatively, the first window 901 is hidden when the phone detects a voice command indicating that the first window should be hidden. Or, the first window 901 is hidden when the phone detects that the user clicks on an area of ​​the second interface that is not covered by the first window 901.

[0141] After the first window 901 is hidden, it can still be brought back up. For example, after the first window is hidden, it can be brought back up using a bring-up command. For instance, when the phone detects an operation on the close button 903 in the first window 901, the first window 901 is hidden, and icon 1001 appears on the phone screen. Please refer to [link to relevant documentation]. Figure 10 In (a) of the diagram, when an operation on icon 1001 is detected, the first window 901 is invoked. Of course, icon 1001 can also be displayed in other locations, such as... Figure 10 In the status bar of (b) or, displayed in Figure 10 In the notification bar interface of (c) in the middle, or displayed in the downward slide notification bar interface. Figure 10 In (d) of the control center interface, pull up the menu. Alternatively, when a voice command to bring up the first window is detected, bring up the first window.

[0142] Therefore, in the first method above, the first window 901 automatically appears on the screen at the same time the phone projects the first window, which is convenient for users and provides a better user experience.

[0143] The second method

[0144] When the phone casts the first application to the target device, the phone displays the interface of the second application and shows control buttons. When an operation on the control buttons is detected, the first window is displayed.

[0145] For example, please see Figure 11 In (a), when the phone detects an action to select a TV icon from the target device list, it casts the first application to the TV. For example... Figure 11 In (b) of the example, the interface of the first application is displayed on the television screen. For example... Figure 11 In step (c), after the first application is launched, the phone displays the interface of the second application, along with control button 1101. When an operation on control button 1101 is detected, the phone displays the first window 901, as shown below. Figure 11 (d) in the middle.

[0146] Among them, the control button 1101 can be displayed in any position, for example Figure 11In (c), the control button 1101 is displayed floating on the interface of the second application, or it can be displayed in the status bar, or in the downward notification bar interface, or in the upward control center interface, etc. This application embodiment does not limit the scope.

[0147] In the second method, the display position and / or display area of ​​the first window 901 can be set by default or adjustable by the user. Furthermore, the first window can also be hidden, and it can be brought back up after being hidden. The implementation principle is the same as the first method, and will not be repeated here.

[0148] Therefore, the second method ( Figure 11 (as shown) and the first method above ( Figure 9 The difference between the two methods is as follows: In the first method, the first window 901 automatically appears on the phone screen when the phone casts the first application to the target device; in the second method, the control button 1101 automatically appears on the phone screen when the phone casts the first application to the target device, and the first window 901 appears when an operation on the control button 1101 is detected.

[0149] Optionally, the mobile phone may use the first method or the second method by default (e.g., the system default), or the user may set the first method or the second method to be used. This application embodiment does not limit this.

[0150] Users can control the application projected onto the target device through the first window 901.

[0151] For example, please see Figure 12 (a) and Figure 12 In (b), the interface displayed in the first window 901 on the mobile phone is the same as the display interface of the first application projected from the target device; both are interface A of the first application. Figure 12 (c) and Figure 12 In step (d), when the user controls the first application within the first window 901, causing the first application to update to interface B, the corresponding first application projected onto the target device also updates its interface B. Assuming the first application projected onto the target device is a photo album application, the process of the user controlling the photo album application projected onto the target device through the first window can be found in the previous text. Figure 4B Assuming the primary application cast to the target device is a video playback application, please refer to the previous section for the process of controlling the video playback application cast to the target device through the first window. Figure 5B .

[0152] Figure 13 The background processing flow for displaying the first window (901) on the mobile phone. For example... Figure 13The phone projects its screen to the target device's application, thus creating a first Visual Display (VD). The phone also creates a second VD (optionally, the display parameters of the second VD can be adapted to the phone's local display parameters, such as having the same resolution). The display information of the first application in the first VD can be mapped (e.g., mirrored) to the second VD. The phone also creates a drawing surface to hold the display information in the second VD; in other words, the display information in the second VD is displayed through the Surface. Simply put, the second VD is mapped to the local display (i.e., mapped to the first window for display) via the surface.

[0153] Figure 13 The first VD sends the display information of the first application to the second VD and the target device respectively. Intuitively, the first VD acts as a data source, providing the display information of the first application to both the left and right directions. In this way, the first application always runs in the first VD, meaning that application casting is always in progress and never terminated, so there is no need to rebuild the mirror casting (i.e., there is no need to switch the application casting to mirror casting). In other words, the establishment of the second VD and the first window (i.e., the surface corresponding to the second VD) does not affect the application casting process.

[0154] Figure 13 In the process, when the interface of the first application in the first window (i.e. the surface corresponding to the second VD) on the local display screen is updated (for example, when the user controls the first application in the first window, causing the display interface of the first application to be updated), the interface of the first application in the second VD will be updated accordingly. Consequently, when the interface of the first application in the first VD is updated, the interface of the first application on the target device will also be updated.

[0155] It should be noted that the previous Figure 3 As mentioned earlier, when casting an application to a target device, you need to switch the application casting to mirroring mode. Below, we'll combine... Figure 3 and Figure 13 This paper explains the differences between the current solution and the technical solution provided in the embodiments of this application.

[0156] Figure 3 The process of switching from app casting to mirror casting involves: First, the first app in the first VD (Virtual Display) is rolled back to the local display. After this rollback, the first VD no longer contains the first app, and app casting terminates. Then, mirror casting is established, mirroring the first app from the local display onto the first VD. The display information in the first VD is synchronized to the target device. In short, app casting needs to be terminated first, and then mirror casting needs to be re-established to control the app cast to the target device. Figure 13In this process, the first VD acts as the data source, sending the display information of the first application to both the second VD and the target device. Therefore, the first application always runs on the first VD, so application mirroring does not need to be terminated or the mirroring process needs to be re-established, which is more efficient than traditional mirroring. Figure 3 The process is simplified.

[0157] Figure 14 This is a flowchart illustrating a method for controlling the display interface projected onto a target device, as provided in an embodiment of this application.

[0158] like Figure 14 As shown, the process of the method includes:

[0159] S1, the phone displays the interface of the first application.

[0160] S2, the phone receives a screen mirroring request and determines whether it is in mirrored screen mirroring mode.

[0161] For example, after the phone receives a screen mirroring request, it can output a prompt message to remind the user to choose between mirrored screen mirroring mode or non-mirrored screen mirroring mode (i.e., application screen mirroring mode).

[0162] S3, if not in mirroring mode, creates the first VD.

[0163] S4 moves the first application from the local display to the first VD for execution.

[0164] The term "moving" here can be understood as moving the running task of the first application from the task stack of the local display to the task stack of the first VD; it can also be called "stack shifting." After the first application is moved from the local display to the first VD for execution, the local display no longer has the running task of the first application.

[0165] S5, the first VD projects the first application onto the target device for display.

[0166] S6 determines whether to control the first application projected onto the target device based on the received input operation.

[0167] For example, the input action is clicking. Figure 11 The operation of the control button 1101 (corresponding to the second method above); or, when a voice command is given to control the first application cast to the target device, it is determined that the first application cast to the target device should be controlled.

[0168] Optionally, S6 can be omitted, so S6 is represented by a dashed line in the diagram. If S6 is not executed, the process proceeds directly from S5 to S7 to create the second VD. This can be understood as the first window automatically appearing in the foreground of the main device after the first application is deployed (corresponding to the first method mentioned earlier). Optionally, S6 can be executed immediately after S5, or after S7 or S8; this embodiment does not impose any limitations.

[0169] S7, if so, create a second VD.

[0170] S8, establish a mirror mapping relationship between the first VD and the second VD. The mirror mapping relationship is used to indicate the mirror mapping of the first VD to the second VD.

[0171] For example, prior to S8, the steps could include: setting a first identifier for the first VD and a second identifier for the second VD. The first identifier and / or the second identifier can be either a VD name or an ID, without limitation. For instance, the first identifier of the first VD could be a source VD, and the second identifier of the second VD could be a target VD, used to indicate that the first VD is mirrored to the second VD.

[0172] Optionally, prior to S8, a step may be included: setting a flag that indicates whether control is needed for the first application cast to the target device. For example, a flag of 1 indicates that control is needed for the first application cast to the target device, while a flag of 0 indicates that no control is needed for the first application cast to the target device.

[0173] S9, based on the mirror mapping relationship, the first application in the first VD is mapped to the second VD, and the second VD maps the first application to the surface corresponding to the second VD.

[0174] Understandably, prior to the S9, the process could include creating a surface, which could be a display area on the local screen, such as the first window. Of course, the surface could also be something that already exists at the factory, for example, something that's pre-existing on the phone; in that case, creating a surface wasn't necessary before the S9.

[0175] S10 updates the interface of the first application on the Surface based on user actions.

[0176] S11, the updated interface in the surface is mapped to the second VD, the second VD maps the updated interface to the first VD, and the first VD projects the updated interface to the target device.

[0177] Through the above steps, users can control the applications projected onto the target device via the surface (i.e., the first window) corresponding to the second VD.

[0178] In this embodiment of the application, controlling the display interface projected onto the target device by creating a second VD has the following beneficial effects:

[0179] First, it should be noted that, generally, mobile phone systems have a built-in mirror mapping function, meaning that the mirror mapping function can be used directly without modifying the underlying code. In this embodiment, the creation of a second VD and the mapping from the first VD to the second VD can use the system's built-in mirror mapping, thus avoiding modifications to the underlying code. Moreover, the mapping from the second VD to the local display on the phone is a surface mapping, which also does not require underlying processing. Therefore, the technical solution provided in this embodiment has low implementation difficulty and high processing efficiency as it does not require underlying processing.

[0180] Optionally, the phone can also avoid creating a second VD. In this case, the first VD directly maps its display information to the first window of the local display screen for display, which is a more direct approach. When mapping the first VD directly to the local display screen, mirror mapping can be avoided. Instead, it undergoes low-level processing: the first VD's display information is rendered and composited at the lower level before being displayed on the local display screen (e.g., the first window). For example, a new surface can be created for the first VD (because a surface was already created when the first VD was projected onto the target device to carry the displayed information; this surface is called the input surface), so the first VD corresponds to two surfaces. In this case, the display information of the input surface can be mapped to the newly created surface. Although this method requires modification to the underlying code, it avoids creating a second VD, saving VD creation costs.

[0181] Example 2

[0182] This second embodiment can be understood as a refinement of the first embodiment. This second embodiment describes the information interaction between different modules within the main device, the information interaction between different modules within the target device, and the information interaction process between the main device and the target device.

[0183] Please see Figure 15 This is a flowchart illustrating a method for controlling an application projected onto a target device in a projection scenario, as provided in this application embodiment. This flowchart can be understood as... Figure 14 A more detailed breakdown of the flowchart shown. Specifically, Figure 15Sections S1501 to S1508 describe the implementation process of screen mirroring. Sections S1509 to S1517 describe the implementation process of application screen mirroring, which can be understood as... Figure 14 The details of S2 to S5 are further elaborated. S1518 to S1531 describe the process of controlling the display interface on the target device when casting the application, which can be understood as... Figure 14 The refinement process from S6 to S11.

[0184] like Figure 15 The process includes:

[0185] S1501, the mirroring module in the main device receives the mirroring command.

[0186] S1502, the mirroring module creates the first VD.

[0187] S1503, the screen mirroring module maps the display information on the host device's local display screen to the first VD.

[0188] S1504, the mirroring module sends the display information in the first VD to the projection encoding module.

[0189] S1505, the screen projection encoding module encodes the displayed information.

[0190] For example, encoding methods include H.264, MPEG-4, etc., and the embodiments of this application are not limited to these.

[0191] S1506, the screen projection encoding module sends the encoded display information to the target device.

[0192] S1507 After receiving the encoded display information, the target device decodes it through the screen projection decoding module.

[0193] S1508, the screen projection decoding module in the target device sends the decoded display information to the screen projection surface module for display.

[0194] The above process describes screen mirroring. The application mirroring process includes the following steps:

[0195] S1509, The application projection module in the main device receives the application projection command.

[0196] S1510, the application projection module sends a VD creation request to the mirror projection module.

[0197] S1511, the mirroring module creates the first VD.

[0198] S1512, the application projection module moves the first application to the first VD to run.

[0199] The term "moving" here can be understood as moving the running task of the first application from the task stack of the local display to the task stack of the first VD; it can also be called "stack shifting." After the first application is moved from the local display to the first VD for execution, the local display no longer has the running task of the first application.

[0200] S1513, the mirroring module sends the display information in the first VD to the projection encoding module.

[0201] S1514, the screen projection encoding module encodes the displayed information.

[0202] S1515, the screen projection encoding module sends the encoded display information to the target device.

[0203] S1516 After receiving the display information with the number, the target device decodes it through the screen projection decoding module.

[0204] S1517, the screen projection decoding module in the target device sends the decoded display information to the screen projection surface module for display.

[0205] When casting to an application, the process by which the main device controls the display interface cast to the target device includes:

[0206] S1518, the application projection module sends the received control command to the application projection surface module. The control command is used to instruct the control of the display interface projected onto the target device.

[0207] S1519, Use the screen mirroring surface module to create a second VD.

[0208] S1520, the application projection surface module sends a mirror mapping instruction to the mirror projection module, which instructs the mirror projection module to map the display information in the first VD to the second VD.

[0209] S1521, the mirroring module maps the display information in the first VD to the second VD.

[0210] S1522, The application uses the screen mirroring surface module to create a surface to carry the display information in the second VD.

[0211] S1523, the application projection surface module maps the display information in the second VD onto the surface for display.

[0212] S1524, the application projection surface module receives an input operation, which is used to update the display information on the surface.

[0213] S1525, the application projection surface module updates the display information in the second VD.

[0214] S1526, The application projection surface module synchronizes the updated display information in the second VD to the first VD.

[0215] S1527, the first VD sends the updated display information to the screen casting encoding module.

[0216] S1528, the screen projection encoding module encodes the updated display information.

[0217] S1529, the screen projection encoding module sends the encoded display information to the target device.

[0218] S1530: After receiving the display information with the number, the target device decodes it through the screen projection decoding module.

[0219] S1531, the screen projection decoding module in the target device sends the decoded display information (updated display information) to the screen projection surface module for display.

[0220] Example 3

[0221] This third embodiment can be understood as a refinement of the first embodiment. This third embodiment describes the information interaction between different modules within the main device, as well as the information interaction process between different modules in the main device.

[0222] Please see Figure 16 This is a functional block diagram of the main device provided in the embodiments of this application. This diagram can also be understood as a software architecture diagram of the main device. Figure 16 The main device includes four modules: display manager, display manager service, VD manager, and surface manager. These four modules can reside in the main device's Framework layer, which is located below the application layer.

[0223] The Display Manager is responsible for connecting the upper layer (such as the application layer) and the ROM logic layer. It can be understood as a bridge between the upper layer and the ROM logic layer, and can perform some logical processing on the upper layer data. For example, the Display Manager can provide an upper layer interface that can be called by applications in the application layer to implement dual VD mapping.

[0224] The display management service module is responsible for the mapping logic processing of dual VDs (e.g., establishing the mapping relationship between the source VD and the destination VD), and can also be responsible for determining the DF of the VD. The display management service can be located in the Framework, and the scheme responsible for dual VD mapping is implemented in the Framework.

[0225] The VD Manager is responsible for managing VDs, such as creating and deleting them. It can also set the MirrorFlag for VDs and receive calls from the application layer to create VDs for application projection.

[0226] The surface manager is responsible for displaying images on the surface.

[0227] Please see Figure 17 This is a flowchart illustrating a control method for projecting an application onto a target device according to an embodiment of this application. This diagram can be understood as... Figure 16 A schematic diagram illustrating the information interaction between different modules in the main equipment. Specifically, Figure 17 Sections S1701 to S1704 describe the application casting process, which can be understood as... Figure 14 The details of S2 to S5 are further elaborated. S1705 to S1715 describe the process of controlling the display interface on the target device when casting the application, which can be understood as... Figure 14 The refinement process from S6 to S11.

[0228] like Figure 17 As shown, the process includes:

[0229] S1701, after receiving the application casting instruction, the application layer sends a VD creation request to the VD manager.

[0230] This can be understood as the application layer calling the VD manager to create VDs.

[0231] S1702, VD Manager creates the first VD.

[0232] S1703, the display management service module moves the application to be projected (such as the first application) to the first VD to run.

[0233] S1704, VD Manager sets the first screen flag (DISPLAY_FLAG, DF) for the first VD.

[0234] Generally, a VD's DF (Display Defender) can be flag 1. For example, flag 1 could be "OWN_CONTENT_ONLY" (Display only its own content). This flag 1 indicates that only the content contained within the VD will be displayed. If the first application is running on the VD, then the interface of the first application will be displayed (i.e., application mirroring). If no application is running on the VD, a black screen will be displayed. Alternatively, a VD's DF can be flag 2. For example, flag 2 could be "AUTO_MIRROR" (Automatic Mirroring). This flag 2 indicates that when no application is running on the VD, the default display interface of the main screen can be mirrored. Therefore, in S1703, the first DF of the first VD can be either OWN_CONTENT_ONLY or AUTO_MIRROR.

[0235] For example, the VD manager can use the VD identifier setting (setVdFlag) method to set the first DF of the first VD.

[0236] The above process completes the application casting process on the main device. The control flow for the application cast to the target device during application casting includes the following steps.

[0237] S1705, the application layer sends a dual VD command to the display manager.

[0238] Optionally, the dual VD instruction may include a first identifier of the first VD and a second identifier of the second VD, and may also include other information, such as display parameters of the local display screen.

[0239] For example, the application layer can call the display manager using the `setMirrorDisplay` method to send dual VD commands to the display manager; alternatively, the system provides a `setMirrorDisplay` interface, which the application layer can call to send dual VD commands to the display manager. For example, the input parameters of the `setMirrorDisplay` interface include `(target Display Name, source Display Id)`, where `targetDisplayName` refers to the name of the destination (second VD) in the dual VD mapping, and `source Display Id` refers to the unique identifier of the source (first VD) in the dual VD mapping. Therefore, when the application layer calls the `setMirrorDisplay` interface, it needs to input the identifiers of the first and second VDs.

[0240] S1706, The display management service module obtains the first identifier of the first VD and the second identifier of the second VD.

[0241] Optionally, the display management service module can obtain the first identifier of the first VD and the second identifier of the second VD in the dual VD mapping instruction. Alternatively, the display manager can inject the destination (i.e., the second identifier of the second VD) into the display management service module using the `setTargetDisplayName` method and inject the source (i.e., the first identifier of the first VD) into the display management service module using the `setSourceDisplayId` method. Once the first identifier of the first VD and the second identifier of the second VD are injected into the display management service module, they can be used by other modules in the Framework layer.

[0242] S1707, The display management service module sends a VD creation request to the VD manager, which carries the second identifier of the second VD.

[0243] S1708, VD Manager creates a second VD.

[0244] S1709, the VD manager sets the second DF of the second VD. The second DF is used to indicate that the second VD is used to control the application casting on the target device; or, it is used to indicate that the source of the second VD is a specific VD (i.e., the first VD).

[0245] In addition to the above-mentioned markers 1 and 2, this application also provides a marker 3, which can be called custom mirror display (MIRROR_CUSTOM_DISPLAY), indicating that the second VD is used to control the display information projected onto the target device; or, it is used to indicate that the source of the second VD is a specific VD (i.e., the first VD).

[0246] For example, the VD manager can set the second DF using the setVdFlag method.

[0247] For example, after setting the second DF, the second DF will take effect at the staking point in the display management service module, which will be described in detail later.

[0248] S1710, the display management service module sends a surface creation request to the interface manager to request the creation of a surface to host the second VD.

[0249] S1711, the interface manager creates the surface corresponding to the second VD.

[0250] For example, the interface manager can use the setVdSurface method to create an independent drawing surface to carry the display information of the second VD, that is, to draw the display information in the second VD onto the surface for display.

[0251] It should be noted that the execution order of S1707 to S1709 and S1710 to S1711 is not limited in this application embodiment. The second VD can be created first and then the surface corresponding to the second VD can be created; or the surface can be created first and then the second VD corresponding to the surface can be created.

[0252] S1712, the display management service module determines, based on the second DF, whether it needs to control the application projected to the target device; or, it needs to map the display information in a specific VD to the second VD.

[0253] For example, the display management service module can use the `configure Display Locked` method to stub a different DF than the two DFs mentioned above (marked 1 and marked 2), namely mark 3, which is `MIRROR_CUSTOM_DISPLAY`. After the VD manager sets the second DF of the second VD to `MIRROR_CUSTOM_DISPLAY` in S1710, the second DF takes effect at the stub point. This can be understood as the display management service module identifying the stub point; when it identifies the second DF as `MIRROR_CUSTOM_DISPLAY`, it determines whether it needs to control the application projected onto the target device or whether it needs to map display information from a specific VD to the second VD. For example, the display management service module can run the `isMirrorFlag` method to determine the stub point.

[0254] S1713, Establish the mirror mapping relationship between the first VD and the second VD.

[0255] For example, when the display management service module determines that the second DF is MIRROR_CUSTOM_DISPLAY, it can insert the first identifier of the first VD at the stub point to indicate that there is an association between the stub point and the first VD, that is, the second VD and the first VD have established a mirror mapping relationship. For example, the display management service module can use the getMirrorDisplay method to establish a mirror mapping relationship at the stub point based on the first identifier of the first VD and the second identifier of the second VD.

[0256] S1714, replace the display information that was originally to be mapped to the second VD with the display information in the first VD.

[0257] It should be noted that after creating the second VD in S1708, if no application is running in the second VD, it will default to mapping the current display information on the main screen. In this embodiment, since the second DF of the second VD is set to MIRROR_CUSTOM_DISPLAY, and a mirror mapping relationship between the first VD and the second VD is established, the display information that was originally to be mapped to the second VD (i.e., the current display information of the main screen) is replaced with the display information in the first VD, thus realizing the mapping from the first VD to the second VD.

[0258] For example, S1714 can be completed jointly by the display management service module and the VD manager. For instance, the display management service module sends an instruction to the VD manager to instruct the display information in the first VD to be mapped to the second VD, and then the VD manager executes the instruction.

[0259] S1715, the VD manager sends the display information from the second VD to the interface manager so that it can be displayed through the surface corresponding to the second VD created in the interface manager.

[0260] Example 4

[0261] In the previous embodiment one, the display information in the second VD was displayed through the first window on the main device, so the main device controlled the display information projected onto the target device through the first window. In this embodiment four, it is considered that if the display information in the second VD on the main device is displayed through another device, then the display information projected onto the target device can be controlled through that other device. Simply put, the main device transfers control of the display information projected onto the target device to another device. In this way, the main device can perform other operations without interruption, while the other device controls the display information projected onto the target device.

[0262] For example, please see Figure 18 This is a schematic diagram of an application scenario in this embodiment four.

[0263] like Figure 18 In (a) of the diagram, the main device projects the first application onto both target device 1 and target device 2, meaning that the same projected interface is displayed on both target device 1 and target device 2. After the first application is projected, the main device displays the interface of the second application, such as... Figure 18 (b) In this context, the user can control the display information projected from the main device to the target device 2 from the target device 1. For example, ... Figure 18 In (b), when the screen mirroring interface on target device 1 is updated from interface A of the first application to interface B of the first application, the screen mirroring interface on target device 2 is also updated to interface B of the first application.

[0264] In this scenario, target device 1 and target device 2 can be the same type of device or different types of devices. For example, target device 1 can be a mobile phone, PC, or other portable device, while target device 2 can be a television or other similar device. An exemplary scenario is that the main device is mobile phone A, target device 1 is mobile phone B, and target device 2 is a television. A first application on mobile phone A is projected onto mobile phone B and the television. The user of mobile phone A can perform other operations on mobile phone A, and the user of mobile phone B can control the information displayed on the television screen projected from mobile phone A using mobile phone B.

[0265] Figure 18 Please refer to the corresponding main device backend processing flow. Figure 19 As shown. Figure 19 The master device creates a first VD and a second VD. The first application, displayed on the local screen, is moved to the first VD and run. The first VD then projects the first application onto the target device 2. The display information in the first VD is also mapped to the second VD, which projects the display information onto the target device 1. In other words, the first VD corresponds to the target device 2, and the second VD corresponds to the target device 1.

[0266] When the interface of the first application on target device 1 is updated, the display information in the second VD is updated. The second VD maps the updated display information to the first VD, and the first VD projects the updated display information onto target device 2. That is, target device 1 controls the display information projected from the main device onto target device 2.

[0267] Please combine Figure 13 and Figure 19 Therefore, Figure 13 In this context, the second VD corresponds to the local display screen, so the display information projected onto the target device can be controlled through the local display screen. Figure 19 In this context, the second VD does not correspond to the local display screen, but to the target device 1. Therefore, the display information projected from the main device to the target device 2 can be controlled through the target device 1.

[0268] One possible approach is for the main device to modify the original correspondence between the second VD and the local display screen to a correspondence between the second VD and the target device 1. In this way, the main device no longer needs to display the display information in the second VD, i.e., no longer needs to display the first window, while the target device 1 displays the display information in the second VD.

[0269] Based on the same inventive concept, this application provides a screen projection control method. This method can be applied to a communication system, which includes a first electronic device and a second electronic device. The first electronic device can be the master device mentioned above, and the second electronic device can be the target device mentioned above. The master device is the sending end in the screen projection scenario, and the target device is the receiving end in the screen projection scenario. Of course, the roles of the sending end and the receiving end can be interchanged; that is, the sending end can project its screen to the receiving end, and correspondingly, the receiving end can also project its screen to the sending end. For example, the first electronic device can be a mobile phone, tablet computer, PC, watch, or other device with a display screen. The target device can be a tablet computer, PC, television, or other large-screen device.

[0270] like Figure 20 The method includes the following steps:

[0271] S2001, The first electronic device displays the first interface of the first application.

[0272] For example, with Figure 7 (a) and Figure 7 Taking (b) as an example, when the first electronic device detects an operation to open the first application (e.g., clicking the icon of the first application on the main interface), the first electronic device displays the first interface of the first application. The first application can be any application, for example, it could be... Figure 4A The photo album application in (a) of the text, or, is Figure 5A The video playback application in (a) and so on.

[0273] S2002, the first electronic device receives the screen projection operation.

[0274] For example, the screen casting operation may include swiping up from the bottom edge of the display screen and tapping the screen casting icon in the swipe-up menu (which appears in response to the swipe-up operation); or, it may include swiping down from the top status bar of the phone screen and tapping the screen casting icon in the notification panel (which appears in response to the swipe-down operation), etc., or it may be a voice command to indicate screen casting. This application embodiment does not limit the scope of the operation.

[0275] S2003, in response to the screen mirroring operation, the first electronic device sends the display information of the first interface to the second electronic device.

[0276] Optionally, prior to S2003, the process may further include displaying a device list of the first electronic device, and determining the second electronic device based on the user's selection operation in the device list, such as... Figure 7 As shown in (c) in the figure.

[0277] S2004, The second electronic device displays the first interface of the first application.

[0278] For example, such as Figure 7 In (d), the second electronic device displays the first interface of the first application.

[0279] S2005, the first electronic device displays a second interface and a first window of the second application, the first window including a third interface, the third interface being the same as the current display interface of the first application projected onto the second electronic device.

[0280] The first implementation method is to use Figure 9 For example, when the first electronic device launches the first application, it displays the second interface of the second application and automatically displays the first window 901. The second implementation method is as follows: Figure 11 For example, when the first electronic device projects the first application, it displays the second interface of the second application and also displays control buttons 1101. When an operation on control buttons 1101 is detected, the first electronic device displays a first window 901. The information displayed in the first window 901 is the same as the information displayed on the second electronic device, and the user can control the information displayed on the second electronic device through the first window 901. It should be noted that all the descriptions of the first window above apply to this embodiment, and will not be repeated here to save space.

[0281] For the background implementation process of the first electronic device displaying the first window, please refer to [link / reference needed]. Figure 13 I will not repeat myself.

[0282] S2006, the first electronic device receives a user operation.

[0283] For example, the user operation refers to an operation used to update the information displayed in the first window. For instance, such as... Figure 4B The user operation could be opening an image in the photo album application within the first window 401. Or, as... Figure 5B The user operation can be opening the next episode of the video playback application on the TV in the first window 401.

[0284] S2007, in response to the user's operation, the first electronic device updates the third interface in the first window to the fourth interface.

[0285] For example, with Figure 12 For example, under the user's operation, the interface A of the first application in the first window 901 is updated to the interface B of the first application.

[0286] S2008, the first electronic device sends the display information of the fourth interface to the second electronic device.

[0287] S2009, the display interface of the first application on the second electronic device is updated to the fourth interface.

[0288] For example, continue with Figure 12 For example, when the interface A of the first application within the first window 901 is updated to the interface B of the first application, the corresponding interface A of the first application projected onto the second electronic device is also updated to interface B. For the background processing of controlling the information displayed on the second electronic device through the first window, please refer to the previous section on... Figure 13 The relevant descriptions will not be repeated here.

[0289] In some embodiments, the first electronic device can also send the display information of the fourth interface to a third electronic device. The third electronic device displays the fourth interface. That is, the first electronic device projects its screen to both the second and third electronic devices. The first electronic device can transfer control of the display information projected onto the second electronic device to the third electronic device. Thus, when the third electronic device controls the fourth interface on its local display to update to the fifth interface, the second electronic device correspondingly updates the fourth interface on its local display to the fifth interface. For example, with... Figure 18 Taking (a) as an example, the main device projects the first application onto both target device 1 and target device 2, and transfers control of the displayed information on target device 2 to target device 1. Therefore, Figure 18 In (b) of the above, when target device 1 updates the interface of the first application projected onto target device 1 to interface B, the corresponding interface of the first application projected onto target device 2 is also updated to interface B. For the background implementation process of the main device transferring control of the displayed information projected onto target device 1 to target device 2, please refer to [link to relevant documentation]. Figure 19 This will not be repeated here.

[0290] Optionally, after the first electronic device transfers control of the display information projected onto the second electronic device to the third electronic device, the first electronic device may cancel the display of the first window.

[0291] Based on the same concept Figure 21 This is a schematic diagram of the structure of the electronic device 2100 provided in an embodiment of this application. The electronic device 2100 can be the main device or the target device mentioned above. Figure 21As shown, the electronic device 2100 may include: one or more processors 2101; one or more memories 2102; a communication interface 2103; and one or more computer programs 2104. These devices can be connected via one or more communication buses 2105. The one or more computer programs 2104 are stored in the memory 2102 and configured to be executed by the one or more processors 2101. The one or more computer programs 2104 include instructions. For example, these instructions can be used to perform relevant steps of the main device as described in the corresponding embodiments above. The communication interface 2103 is used to enable communication between the main device and other devices (such as target devices); for example, the communication interface may be a transceiver.

[0292] Based on the same concept, this application also provides a communication system. The communication system includes a first electronic device and a second electronic device. The first electronic device can be the main device mentioned above, and the second electronic device can be the target device mentioned above. For example, the first electronic device can be a device with a display screen, such as a mobile phone, tablet computer, PC, or watch. The second electronic device can be a large-screen device, such as a tablet computer, PC, or television. The structures of the first and second electronic devices can be found in [reference needed]. Figure 21 As shown. For example, when Figure 21 When the illustrated electronic device 2100 is the first electronic device, when the instructions of one or more computer programs 2104 are executed by a processor, the first electronic device performs the steps of the first electronic device (i.e., the master device) as described above. Figure 21 When the electronic device 2100 shown is a second electronic device, when the instructions of one or more computer programs 2104 are executed by the processor, the second electronic device performs the steps of the second electronic device (i.e., the target device) as described above.

[0293] In the embodiments provided above, the methods provided by the present application are described from the perspective of an electronic device (e.g., a main device or a target device) as the executing entity. To implement the functions of the methods provided in the embodiments of the present application, the electronic device may include hardware structures and / or software modules, implementing the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is executed in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0294] In the above embodiments, the terms "when..." or "after..." can be interpreted, depending on the context, as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, the phrases "when..." or "if (the stated condition or event) is detected" can be interpreted, depending on the context, as meaning "if...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)". Furthermore, in the above embodiments, relational terms such as "first" and "second" are used to distinguish one entity from another, without limiting any actual relationship or order between these entities.

[0295] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0296] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)). Where there is no conflict, the solutions of the above embodiments can be used in combination.

[0297] It should be noted that a portion of this patent application contains copyrighted material. The copyright holder retains all rights except for making copies of the contents of patent documents or records from the patent office.

Claims

1. A screen projection control method, applied to a system including a first electronic device and a second electronic device, characterized in that, The method includes: The first electronic device displays a first interface of the first application; In response to the screen mirroring operation, the first electronic device creates a first VD and transfers the first application to the first VD for execution. The first VD is used to mirror the interface of the first application to the second electronic device. The second electronic device displays the first interface of the first application; The first electronic device displays a second interface of a second application and a first window. The display interface in the first window is the same as the current display interface of the first application projected onto the second electronic device. The display of the first window includes: creating a second VD, mapping the display interface of the first application in the first VD to the second VD, mapping the display interface in the second VD to the first window for display, and the first window can control the display information projected from the first electronic device onto the second electronic device. In response to a user operation, the first electronic device updates the third interface in the first window to the fourth interface. Correspondingly, the display interface in the second VD is updated to the fourth interface, and the display interface in the first VD is updated to the fourth interface, so that the display interface of the first application on the second electronic device is updated to the fourth interface. The first electronic device modifies the correspondence between the second VD and the first window to a correspondence between the second VD and the third electronic device, so that the display information in the second VD is mapped to the third electronic device for display, and the third electronic device can control the display information projected by the first electronic device onto the second electronic device; When the third electronic device controls the fourth interface to be updated to the fifth interface, the display information in the second VD is updated to the fifth interface, and the fifth interface in the second VD is mapped to the first VD. The first VD is used to project the fifth interface onto the second electronic device. After the first electronic device modifies the correspondence between the second VD and the first window to the correspondence between the second VD and the third electronic device, it cancels the display of the first window.

2. The method according to claim 1, characterized in that, The first window is displayed floating above the second interface, and the first window does not completely cover the second interface; or, the second interface and the first window are displayed in a split-screen format.

3. The method according to claim 1, characterized in that, The first electronic device displays a first window, including: Received an operation to invoke the first window; In response to the operation of bringing up the first window, the first window is displayed.

4. The method according to claim 3, characterized in that, In response to the operation of invoking the first window, displaying the first window includes: The second interface displays the first marker; In response to an operation on the first mark, the first window is invoked.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: In response to the operation of hiding the first window, the first window is hidden; or, In response to an operation for adjusting the display position and / or display area of ​​the first window, the display position and / or display area of ​​the first window are adjusted.

6. The method according to any one of claims 1-4, characterized in that, The first electronic device displays a first window, including: The first electronic device creates a drawing surface, and the second VD is used to map the display interface in the second VD to the first window through the drawing surface.

7. The method according to claim 6, characterized in that, In response to a user action, the first electronic device updates the third interface within the first window to a fourth interface, including: In response to a user operation, the information displayed on the drawing surface is updated to the fourth interface; wherein, the display information of the fourth interface on the drawing surface is mapped to the second VD, the second VD is used to map the display information of the fourth interface to the first VD, and the first VD is used to send the display information of the fourth interface to the second electronic device.

8. A screen projection control method, characterized in that, include: The first electronic device displays the first interface of the first application; In response to the screen mirroring operation, the first electronic device creates a first VD and transfers the first application to the first VD for execution. The first VD is used to mirror the interface of the first application to the second electronic device. The first electronic device displays a second interface of a second application and a first window. The display interface in the first window is the same as the current display interface of the first application projected onto the second electronic device. The display of the first window includes: creating a second VD, mapping the display interface of the first application in the first VD to the second VD, mapping the display interface in the second VD to the first window for display, and the first window can control the display information projected from the first electronic device onto the second electronic device. In response to a user operation, the first electronic device updates the third interface in the first window to the fourth interface. Correspondingly, the display interface in the second VD is updated to the fourth interface, and the display interface in the first VD is updated to the fourth interface, so that the display interface of the first application on the second electronic device is updated to the fourth interface. The first electronic device modifies the correspondence between the second VD and the first window to a correspondence between the second VD and the third electronic device, so that the display information in the second VD is mapped to the third electronic device for display, and the third electronic device can control the display information projected by the first electronic device onto the second electronic device. When the third electronic device controls the fourth interface to be updated to the fifth interface, the display information in the second VD is updated to the fifth interface, and the fifth interface in the second VD is mapped to the first VD. The first VD is used to project the fifth interface onto the second electronic device. After the first electronic device modifies the correspondence between the second VD and the first window to the correspondence between the second VD and the third electronic device, it cancels the display of the first window.

9. The method according to claim 8, characterized in that, The first window is displayed floating above the second interface, and the first window does not completely cover the second interface; or, the second interface and the first window are displayed in a split-screen format.

10. The method according to claim 8, characterized in that, The first electronic device displays a first window, including: Received an operation to invoke the first window; In response to the operation of bringing up the first window, the first window is displayed.

11. The method according to claim 10, characterized in that, In response to the operation of invoking the first window, displaying the first window includes: The second interface displays the first marker; In response to an operation on the first mark, the first window is invoked.

12. The method according to any one of claims 8-11, characterized in that, The method further includes: In response to the operation of hiding the first window, the first window is hidden; or, In response to an operation for adjusting the display position and / or display area of ​​the first window, the display position and / or display area of ​​the first window are adjusted.

13. The method according to any one of claims 8-11, characterized in that, The first electronic device displays a first window, including: The first electronic device creates a drawing surface, and the second VD is used to map the display interface in the second VD to the first window through the drawing surface.

14. The method according to claim 13, characterized in that, In response to a user action, the first electronic device updates the third interface within the first window to a fourth interface, including: In response to a user operation, the information displayed on the drawing surface is updated to the fourth interface; wherein, the display information of the fourth interface on the drawing surface is mapped to the second VD, the second VD is used to map the display information of the fourth interface to the first VD, and the first VD is used to send the display information of the fourth interface to the second electronic device.

15. A communication system, characterized in that, include: First electronic device and second electronic device; The first electronic device includes: a processor; a memory; wherein the memory stores one or more computer programs, the one or more computer programs including instructions that, when executed by the processor, cause the first electronic device to perform the steps of the first electronic device as described in any one of claims 1 to 7; The second electronic device includes: a processor; a memory; wherein the memory stores one or more computer programs, the one or more computer programs including instructions that, when executed by the processor, cause the second electronic device to perform the steps of the second electronic device as described in any one of claims 1 to 7.

16. An electronic device, characterized in that, include: Processor, memory, and one or more programs; The one or more programs are stored in the memory, and the one or more programs include instructions that, when executed by the processor, cause the electronic device to perform the steps of the method as described in any one of claims 8 to 14.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 14.

Images