Screen projection method and electronic equipment

Abstract

The invention provides a screen projection method and electronic equipment, and relates to the technical field of communication and the technical field of terminals. According to the specific scheme, after first equipment projects a screen to second equipment, the second equipment displays a first window; when the first device is in the bright screen state, the second device displays first image information from the first device in the first window, and the first image information corresponds to an application interface of a first application in the first device; after the first device is switched to the screen locking or screen turning-off state, the second device displays second image information from the first device in the first window, and the second image information also corresponds to the application interface of the first application in the first device. Therefore, after the first equipment projects the screen to the second equipment, even if the first equipment is switched from the bright screen state to the screen-off state or the screen-locked state, the second equipment can still normally display the application screen projection window provided by the first equipment.

Description

Screen projection method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 31, 2022, with application number 202211348614.X and invention name “A screen projection method and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the fields of communication technology and terminal technology, and in particular to a screen projection method and electronic device. Background Art

[0003] The collaborative work of multiple electronic devices brings convenience to users' lives and work. Taking collaborative notifications as an example, when a mobile phone receives a message, not only can the message notification be displayed on the mobile phone, but the message notification can also be transferred to the PC for display. This makes it convenient for users to view messages across devices and improves the efficiency of collaborative work between devices. Of course, it is not enough to simply transfer message notifications between devices. After users view the message notifications from their mobile phones on the PC, they often need to process the message notifications through the PC, which requires casting the phone screen to the PC.

[0004] However, during screen mirroring between PC and mobile phone, if the mobile phone enters screen off or lock mode, the PC screen mirroring will be abnormal.

[0005] Summary of the Invention

[0006] An embodiment of the present application provides a screen projection method and electronic device. During screen projection from a first device to a second device, after the first device switches to a locked screen or screen-off state, the second device can still normally display the application projection window provided by the first device.

[0007] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0008] In the first aspect, an embodiment of the present application provides a screen projection method, which is applied to a first device and a second device, and the method includes: after the first device projects the screen to the second device, the second device displays a first window; when the first device is in a bright screen state, the second device displays first image information from the first device in the first window, and the first image information corresponds to the application interface of the first application in the first device; after the first device switches to a locked screen or a screen-off state, the second device displays second image information from the first device in the first window, and the second image information also corresponds to the application interface of the first application in the first device.

[0009] In the above embodiment, the first device projects its screen to the second device in the screen-on state. For example, heterogeneous screen projection is performed. After the heterogeneous screen projection is successful, the first device can draw the application interface of the first application into the first image information, and send it to the second device, which is displayed in the first window by the second device. In this way, the user can use the first application installed in the first device by operating the second device. Afterwards, the first device can enter the screen-off or lock screen state in response to the user's operation, and the first device can draw the second image information based on the application interface of the first application, and send it to the second device, so that the second device can continue to display the application interface related to the first application in the first window. It can be understood that the above-mentioned first image information and second image information both correspond to the application interface of the first application, and can be image information drawn according to the application interface of the first application at different times, that is, the content displayed by the first image information and the second image information can be the same or different.

[0010] This way, after the first device switches to the off or locked screen state, the first device can continue to project its screen to the second device normally. This also prevents excessive energy consumption caused by the screen being on for a long time when the user is not using the first device, and prevents others from using the unlocked first device without the user's permission.

[0011] In some embodiments, after the first device switches to a lock screen or screen-off state, the method further includes: the first device configures the first application to not sleep during its life cycle, and configures the application interface of the first application to be allowed to be drawn in a first display area during its life cycle, where the first display area is an invisible display area in the first device; the first device sends second image information of the first application to the second device, instructing the second device to display the second image information in the first window, where the second image information is data drawn in the first display area.

[0012] In the above embodiment, by changing the first application not to enter sleep mode, the application interface of the first application is kept visible in the foreground, ensuring that after the first device turns off the screen or locks the screen, the user can continue to use the first application in the first device through the first window in the second device.

[0013] In some embodiments, before the first device projects the screen to the second device, the method also includes: the first device receives a first message of the first application; the first device forwards the first message to the second device; the second device displays a first control, which is used to display the first message; the second device sends a first request to the first device in response to the user's operation on the first control, and the first request includes a first identifier of the first application; the first device starts projecting the screen to the second device in response to the first request.

[0014] In some embodiments, after initiating screen projection to the second device, the method also includes: the first device creates a first display area and starts a first application; after the application interface of the first application is drawn in the first display area, the first device generates first image information based on the image data in the first display area; the first device sends the first image information to the second device.

[0015] In some embodiments, before launching the first application, the method further includes: the first device tags the first application with a first tag; before the first device configures the first application to not sleep during its life cycle, and configures the application interface of the first application to be allowed to be drawn in the first display area during its life cycle, the method further includes: the first device determines that the first application has a first tag.

[0016] In some embodiments, before the first device switches to the lock screen or screen off state, the method also includes: the first device responds to the user's instruction to lock the screen or screen off operation, only controls the display screen of the first device to power off or only displays the lock screen layer on the display screen, and the service running status corresponding to the display screen remains unchanged.

[0017] In some embodiments, the first display area is marked with a second label. Before the first device switches to the lock screen or screen off state, the method also includes: when the first device detects that the conditions for automatic screen off are met, determining that the first display area has the second label; the first device only controls the display screen of the first device to power off or only displays the lock screen layer on the display screen, and the service running status corresponding to the display screen remains unchanged.

[0018] In some embodiments, before the first device switches to a lock screen or screen-off state, the method further includes: the first device determining not to draw a lock screen layer in the first display area.

[0019] In some embodiments, before the first device switches to a locked screen or a screen-off state, the method further includes: the first device determining that the first display area does not mirror content in a second display area, where the second display area is a display area visible in the first device.

[0020] In the second aspect, an embodiment of the present application provides a screen projection method, which is applied to a first device, and the method includes: the first device projects the screen to the second device; when the first device is in a bright screen state, the first device sends first image information corresponding to a first application to the second device, instructing the second device to display the first image information on a first window; after the first device switches to a locked screen or a screen-off state, the first device sends second image information corresponding to the first application to the second device, instructing the second device to display the second image information on the first window, and the first image information and the second image information are both application interfaces of the first application.

[0021] In some embodiments, after the first device switches to a lock screen or screen-off state, the method further includes: the first device configures the first application to not sleep during its life cycle, and configures the application interface of the first application to be allowed to be drawn in a first display area during its life cycle, and the first display area is an invisible display area in the first device; the first device generates the second image information based on the content drawn in the first display area.

[0022] In some embodiments, before the first device projects the screen to the second device, the method also includes: the first device receives a first message of the first application; the first device forwards the first message to the second device; after receiving a first request sent by the second device, starting the screen projection to the second device, the first request is a request information for projecting the screen for the first application, and the first request carries a first identifier of the first application.

[0023] In some embodiments, before the first device sends the first image information corresponding to the first application to the second device, the method further includes: the first device creates a first display area and starts the first application;

[0024] After the application interface of the first application is drawn in the first display area, the first device generates the first image information according to the image data in the first display area.

[0025] In some embodiments, before launching the first application, the method further includes: the first device marks the first application with a first tag; before the first device configures the first application not to sleep during its life cycle, and configures the application interface of the first application to be allowed to be drawn in the first display area during its life cycle, the method further includes: the first device determines that the first application has the first tag.

[0026] In some embodiments, before the first device switches to the lock screen or screen off state, the method also includes: the first device responds to the user's instruction to lock the screen or screen off operation, only controls the display screen of the first device to power off or only displays the lock screen layer on the display screen, and the service running status corresponding to the display screen remains unchanged.

[0027] In some embodiments, the first display area is marked with a second label. Before the first device switches to the lock screen or screen off state, the method further includes: when the first device detects that the conditions for automatic screen off are met, determining that the first display area has the second label; the first device only controls the display screen of the first device to power off or only displays the lock screen layer on the display screen, and the service running status corresponding to the display screen remains unchanged.

[0028] In some embodiments, before the first device switches to a lock screen or screen-off state, the method further includes: the first device determines not to draw a lock screen layer in the first display area.

[0029] In some embodiments, before the first device switches to a locked screen or a screen-off state, the method further includes: the first device determining that the first display area does not mirror content in a second display area, where the second display area is a display area visible in the first device.

[0030] In a third aspect, an embodiment of the present application provides an electronic device, which includes one or more processors and a memory; the memory is coupled to the processor, and the memory is used to store computer program code, and the computer program code includes computer instructions. When one or more processors execute the computer instructions, the one or more processors are used to execute the method described in the first aspect and its possible embodiments; or, when one or more processors execute the computer instructions, the one or more processors are used to execute the method described in the second aspect and its possible embodiments.

[0031] In a fourth aspect, an embodiment of the present application provides a computer storage medium comprising computer instructions, which, when the computer instructions are executed on an electronic device, cause the electronic device to execute the method described in the first aspect and its possible embodiments; or, when the computer instructions are executed on an electronic device, cause the electronic device to execute the method described in the second aspect and its possible embodiments.

[0032] It can be understood that the electronic devices, computer storage media and computer program products provided in the above aspects are all applied to the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG1 is a schematic diagram of a message flow scenario provided in an embodiment of the present application.

[0034] Figure 2 is a schematic diagram of the interface display of a heterogeneous screen projection provided in an embodiment of the present application.

[0035] Figure 3 is a schematic diagram of the interface display of a heterogeneous screen projection provided in an embodiment of the present application.

[0036] Figure 4 is a schematic diagram of the principle of heterogeneous screen projection provided in an embodiment of the present application.

[0037] Figure 5 is a schematic diagram of the interface display of a heterogeneous screen projection provided in an embodiment of the present application.

[0038] Figure 6 is a schematic diagram of the interface display of a heterogeneous screen projection provided in an embodiment of the present application.

[0039] FIG7 is a flow chart of the judgment mechanism provided in an embodiment of the present application.

[0040] Figure 8 is a schematic diagram of the interface display of a heterogeneous screen projection provided in an embodiment of the present application.

[0041] FIG9 is a software architecture diagram of an electronic device provided in an embodiment of the present application.

[0042] Figure 10 is a specific implementation process diagram of a screen projection method provided in an embodiment of the present application.

[0043] FIG11 is a specific implementation process diagram of another screen projection method provided in an embodiment of the present application.

[0044] Figure 12 is a specific implementation process diagram of another screen projection method provided in an embodiment of the present application.

[0045] Figure 13 is a specific implementation process diagram of another screen projection method provided in an embodiment of the present application.

[0046] Figure 14 is a specific implementation process diagram of another screen projection method provided in an embodiment of the present application.

[0047] Figure 15 is a specific implementation process diagram of another screen projection method provided in an embodiment of the present application.

[0048] FIG16 is a schematic diagram of a principle provided in an embodiment of the present application.

[0049] FIG17 is another schematic diagram of a principle provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0051] It should be understood that the terms "first," "second," and the like in the specification, claims, and drawings of this application are used to distinguish between different objects, rather than to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0052] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0053] Illustratively, an embodiment of the present application provides a system including multiple electronic devices, such as device 1, device 2, and device 3. The electronic devices in the above system (ie, device 1, device 2, device 3) may belong to the same local area network.

[0054] The local area network may be a wireless communication network, such as a wireless local area network (WLAN), wireless fidelity point to point (Wi-Fi P2P), a Bluetooth network, a ZigBee network, an infrared (IR) or near field communication (NFC) network. Furthermore, the local area network may be a wired communication network, such as a communication network established by a video graphics array (VGA), a digital visual interface (DVI), a high definition multimedia interface (HDMI), or a data transmission line. Furthermore, the local area network may be a remote network.

[0055] After devices 1, 2, and 3 have mutually recognized trust, the system consisting of devices 1, 2, and 3 can be called a trust ring. The process of mutual trust recognition between devices can refer to relevant technologies. For example, the process can be determined based on one or more factors, such as whether the devices have logged into the same system account, whether they have been authorized, and whether near-field communication is used. The details are not detailed here.

[0056] It can be understood that in the same trust ring, multiple electronic devices can safely share data. The shared data includes but is not limited to message reminders, message records, user information of other electronic devices, device types, running status of each application, identification of each application, synchronization of identification of recently used applications, etc.

[0057] In this way, a variety of collaborative services can be realized within the trust ring, such as collaborative calls, collaborative notifications, screen projection and other services.

[0058] Taking collaborative notifications as an example, in a trusted ring, when device 1 receives a message notification, it can not only display it on device 1 but also forward it to device 2 and / or device 3 for display. This makes it easier for users to view messages across devices and improves the efficiency of inter-device collaboration. Similarly, device 2 can forward received message notifications to device 1 and / or device 3, and device 3 can also forward received message notifications to device 1 and / or device 2.

[0059] Of course, it is not enough to just transfer messages between devices. If a user views a message notification from device 1 on device 2, he or she often needs to process the message notification through device 2, which requires casting the screen of device 1 to device 2.

[0060] Generally speaking, a screen projection scenario involves a source device and a target device. The source and target devices communicate via a protocol, enabling the content on the source device to be displayed on the target device. This is known as screen projection. Common protocols include the DLNA (Digital Living Network Alliance) protocol, Chromecast protocol, Miracast protocol, and Airplay protocol. Source devices can be phones, tablets, PCs, watches, and more. Target devices can be tablets, PCs, TVs, and more.

[0061] It is understandable that the source device and the target device may also have other names, for example, the source device may be called an active screen projection device, and the target device may be called a passive screen projection device; or, the source device may be called a first electronic device, and the target device may be called a second electronic device, etc. This application does not limit this.

[0062] As shown in Figure 1, the source device is a mobile phone and the target device is a PC. After the mobile phone and PC are on the same local area network and have established mutual trust, the mobile phone and PC belong to the same trust circle. In other words, a communication channel for message flow can be established between the mobile phone and PC.

[0063] In some embodiments, a mobile phone may have multiple types of applications installed in it. During operation of the mobile phone, the mobile phone may receive application messages pushed by various applications. The application messages come from a message server corresponding to the application.

[0064] For example, a social application may be installed in the mobile phone, and the mobile phone may receive application messages pushed by the social application, such as a message "Are you going for a run today?" sent by Xiao Ming, a contact in the social application.

[0065] After receiving an application message pushed by an application, if the phone is unlocked and the screen is on, the phone can display a prompt message on the current interface. For example, when receiving an application message pushed by a social application, if the current interface of the phone is the main interface 101, a notification bubble 102 is displayed on the main interface 101, and the notification bubble 102 is used to display the application message received by the phone.

[0066] Then, the mobile phone can transfer the above application message (for example, "Are you going for a run today?" pushed by the social application) to the PC through the communication channel used for transferring messages between the mobile phone and the PC.

[0067] In some embodiments, after the PC receives an application message from a mobile phone, it can display a prompt message for the above application message. Among them, the above prompt message can be displayed on the desktop of the PC in the form of a message notification control. For example, the above message notification control can be a floating window, a message bubble, a message card, etc. Of course, in the embodiment of the present application, there is no specific limitation on its display form. For example, as shown in Figure 1, a message card 104 is displayed on the desktop 103, and the message card 104 is used to display the relevant content of the above application message. Among them, the relevant content can be that the application message comes from a mobile phone, or that the application message comes from a social application, or the main content of the application message, etc. It can be understood that in the embodiment of the present application, there is no specific limitation on the display position of the message card 104.

[0068] In the scenario shown in FIG1 , if the user is using a PC, the message card 104 displayed on the desktop 103 can remind the user that the mobile phone has received an application message. In this way, even if the user is not currently using the mobile phone or the mobile phone is not with him, he can still understand the brief content of the application message and avoid missing the message.

[0069] In some embodiments, the message card 104 may have a display time limit. For example, the display time limit is preset to duration 1. In response to the application message forwarded by the mobile phone, the PC begins to display the message card 104. If the display time of the message card 104 exceeds duration 1 and the user does not select the message card 104, the PC may cancel the display of the message card 104.

[0070] For example, when the PC's display screen is a touch screen, the PC detects that the user has made contact with display area 1 of the touch screen, and the PC determines that the user has indicated the selection of message card 104. Display area 1 is the area on the touch screen that displays message card 104. For another example, when a mouse is connected to the PC, the PC detects that the mouse cursor is within display area 1 and a confirmation command is input, and the PC determines that the user has indicated the selection of message card 104.

[0071] In some embodiments, after the PC displays the message card 104, the user can process the application message by operating the PC, or can choose to process the application message by directly operating the mobile phone.

[0072] The method of directly operating the mobile phone to process application messages can refer to related technologies. For example, in response to the user clicking on the notification bubble 102, the mobile phone can run a social application in the foreground and display the application interface containing the above social message. In this way, the user can reply to the application message by operating on the above application interface. In another example, the mobile phone responds to the user's instruction to launch the social application and displays the application interface of the social application. In this way, the user can view the above application message or reply to the application message by operating on the application interface provided by the social application.

[0073] In addition, choosing to operate the PC to process application messages will trigger screen projection between the mobile phone and the PC. For example, the user can select the message card 104 on the PC. After the PC can detect that the user has selected the message card 104, the PC can remotely control the mobile phone to run the social application. After the social application enters the running state, the PC can display the projection window on the desktop 103. After the PC displays the projection window, the user can control the mobile phone through the projection window in the PC, such as operating the mobile phone to process the above-mentioned application messages.

[0074] The above-mentioned screen projection window can be used to synchronously display the current interface of the mobile phone. For example, when the current interface of the mobile phone is the application interface of a social application, the screen projection window also displays the application interface, which is a same-source screen projection. The current interface of the mobile phone can also be displayed asynchronously. For example, when the mobile phone displays the main interface, the screen projection window displays the application interface, which is a different-source screen projection.

[0075] Please refer to Figure 2, which introduces the interface display of heterogeneous screen projection in the above scenario.

[0076] As an implementation method, after the PC determines that the user has selected the message card 104, heterogeneous screen projection can be achieved through data exchange between the PC and the mobile phone. As shown in Figure 2, after the PC determines that the user has selected the message card 104, heterogeneous screen projection is initiated. The main interface 101 is still displayed on the mobile phone's display, and the projection window 201 on the PC's display displays the application interface 202 corresponding to the above application message.

[0077] After that, the content displayed in the projection window will not change synchronously with the content displayed on the mobile phone display. For example, referring to Figure 3, the mobile phone display switches to displaying the information application interface 303, while the projection window 201 on the PC display still displays the application interface 201 corresponding to the above application message.

[0078] Please refer to Figure 4, which is a schematic diagram of the principle of heterogeneous screen projection in Figures 2 and 3 above. In the heterogeneous screen projection scenario, when the source device projects the screen to the target device, the display information on the target device is not synchronized with the display information on the foreground of the source device. The window of application B is displayed on the source device. When the source device projects the window of application B to the target device, the window of application B is displayed in the created virtual display area (VirtualDisplay). Among them, the content of the virtual display area (such as the first display area) is invisible to the user and can be used for projection to the target device for display. In addition to the virtual display area, the source device also includes a user-visible display area (such as the second display area). When the source device projects the window of application B to the target device, the user-visible display area displays the window of application A, that is, the source device displays the window of application A in the foreground. Assume that the user-visible area on the source device switches from the window of application A to the window of application C, that is, the source device displays the window of application C in the foreground. At this time, the window of application B is still displayed on the target device.

[0079] It can be understood that compared with the same-source screen projection, the heterogeneous screen projection method has better isolation. For example, the heterogeneous screen projection method can provide users with independent control screens (i.e., the display screen of the source device and the display screen of the target device) to handle different applications. In this way, after the user projects the window of application B to the target device for display, the source device operates application A, or switches the window of application A to the window of application C, and the user can still operate application B on the target device. Therefore, heterogeneous screen projection releases the control of the display information of the source device over the display information of the target device, thereby enhancing the reusability of multiple devices and allowing users to operate different applications on different devices through projection. In the same-source screen projection method, the content displayed in the projection window changes synchronously with the content displayed on the mobile phone display. In the embodiment of the present application, the heterogeneous screen projection scenario is described as an example.

[0080] Taking Figures 1 and 2 above as an example, it describes that after the mobile phone receives a message notification, it transfers it to the PC. Then, the user operates the PC to select the message card that displays the message notification, triggering the process of heterogeneous screen projection from the mobile phone to the PC.

[0081] During heterogeneous screen projection, as shown in Figure 5, while the PC displays the projection window, the phone is also in the bright and unlocked state. In this scenario, the user interacts with the social application on the phone through the projection window on the PC. In other words, the user can use both the PC and the phone simultaneously by simply operating the PC. While the user is operating the PC, there is no direct interaction between the user and the phone, while the phone is in the bright and unlocked state for a long time. This not only consumes power but also easily leads to privacy leakage. In this case, the user often chooses to click the lock screen button to instruct the phone to enter the screen-off state. In addition, if the phone does not detect user operation for a long time, it will automatically enter the lock screen state. In a related embodiment, during the period when the phone is projecting the screen to the PC, that is, after the PC has normally displayed the projection window (the projection window contains the application interface being projected), if the phone, as the source device, enters the screen-off state, it will directly affect the display of the projection window on the PC. As shown in Figure 5, after the phone enters the screen-off state, the phone's display screen will go black, and at the same time, the projection window on the PC will also go black, which will directly affect the user's ability to remotely control the phone through the PC. It should be noted that the above-mentioned black screen refers to a state where the display screen is not lit or only displays the screen saver image.

[0082] During cross-screen projection, as shown in Figure 6, the PC displays the projection window while the phone is also unlocked and on screen. If the phone receives a lock screen instruction from the user, as shown in Figure 6, the phone enters the lock screen state and displays the lock screen layer. At the same time, the projection window on the PC will also display the lock screen layer, which will directly affect the user's ability to remotely control the phone through the PC.

[0083] In order to solve the above problems, when the source device is locked or in the off state, the user can continue to enable the application projection window on the target device. The embodiment of the present application provides a projection method and electronic device to provide users with a convenient and efficient projection experience.

[0084] Taking the case where the source device is a mobile phone and the target device is a PC as an example, as shown in FIG1 , after the PC receives the application message transferred from the mobile phone, the corresponding message card 104 can be displayed on the main interface 103. Of course, the PC can also receive one or more application messages transferred from any device in the same trust ring. For example, the PC can respond to user operations and display an information list window, which can centrally display received message notifications, or centrally display received and unprocessed message notifications. As an implementation method, the above-mentioned message notifications can also be displayed in the information list window in the form of message cards.

[0085] In some embodiments, if the PC detects a specified operation, such as a user's instruction to display a message list window, the PC may display a message list window on the main interface. This message list window may display application messages from different devices. It is understood that the application messages displayed in the message list window are all messages recently received by the PC from other devices in the trust circle.

[0086] Similar to the message cards displayed on the desktop, when the PC detects that the user has selected any message card in the information list window, it can trigger the device corresponding to the message card to project the screen. For example, the application message displayed on message card a comes from a mobile phone in the trusted ring. When message card a is selected, it can trigger the mobile phone to project the screen to the PC.

[0087] Of course, even after the PC instructs the corresponding device to cast the screen, the casting may fail. For example, if the application corresponding to the message card is protected by an app lock, the casting may fail. Another example is when the channel used for casting is occupied by other devices, resulting in a casting failure. Another example is when the network is unstable, resulting in a casting failure.

[0088] First of all, application lock refers to the protection set by the user of the source device for a certain application in order to protect the security or privacy of the application, and the user needs to enter a pre-set password or fingerprint to open the application. Out of consideration for the security and privacy of the source device, in some possible embodiments, before the actual screen projection, it will be determined whether the application a (for example, the application corresponding to the message card selected by the user on the PC) is optionally protected by the source device. If the source device sets an application lock for application a, it means that the owner of the source device hopes that the application a needs to enter a password or fingerprint to unlock it every time it is accessed. In this case, it is not suitable for such applications with strong privacy protection to start the screen projection directly on the target device side when the source device is in the locked screen state, which goes against the original intention of the user to set the application lock.

[0089] Exemplarily, after the PC detects that the user has selected a message card, it sends a query message 1 to the device corresponding to the message card (such as a mobile phone). The query message 1 is used to query whether the mobile phone has configured an application lock for the message application. If the mobile phone has configured an application lock for the message application, the mobile phone will send feedback information 1 to the PC. The feedback information 1 indicates that the message application is currently protected by an application lock. In this way, after receiving the feedback information 1, the PC can display a reminder message 1. The reminder message 1 is used to remind the user that the application corresponding to the selected message card is protected by an application lock. For example, it can be the text "The message application is protected by an application lock, please go to the mobile phone to unlock the application lock." In addition, the reminder message 1 can be displayed on the PC in the form of a floating window or a card. The embodiment of the present application does not limit the display form of the reminder information.

[0090] Secondly, the source device and the target device can establish a screen projection connection through WLAN or WI-FI P2P. The establishment of the screen projection connection can refer to the method of establishing a communication connection between the source device and the target device above, which will not be repeated here. For the convenience of description, the following embodiment establishes a screen projection connection in WI-FI P2P mode.

[0091] In some embodiments of the present application, after the PC detects that the user has selected a message card, it determines whether the screen projection communication channel (Wi-Fi P2P channel) between the source device (i.e., mobile phone) and the target device (i.e., PC) corresponding to the message card has been occupied by other services. Among them, the source device and the target device establish a screen projection connection through the screen projection channel for data transmission, thereby realizing content mirroring between two or more electronic devices. When the source device and the target device have established other services based on WI-FI P2P, such as PC collaboration, device sharing, etc., the screen projection channel corresponding to the source device and the opposite device is occupied, resulting in the source device being unable to project the screen to the target device based on WI-FI P2P.

[0092] For example, if the PC determines that the screen projection channel (Wi-Fi P2P) between the PC and the mobile phone is occupied, the screen projection fails to start, and a reminder message 2 is displayed. This reminder message 2 exemplarily prompts the user to "screen projection connection conflict, please disconnect the connection with other devices and try again", thereby prompting the user to go to the source device (mobile phone) or target device (PC) to disconnect the screen projection connection with other devices. Among them, the reminder message 2 can be displayed on the PC in the form of a floating window or card. The embodiment of the application does not limit the display form of the reminder message.

[0093] Again, in some embodiments of the present application, after the PC detects that the user has selected a message card, it determines whether the network status of the source device (i.e., the mobile phone) corresponding to the message card meets the screen projection requirements. Among them, the network status includes the flow table matching status, the data packet execution port load status, the total number of lost / forwarded / received data packets or the failure of the data packet TTR and other similar situations. As long as any network status that may occur in the network is included, it is not limited here. It can be understood that the network status will affect the service quality of the screen projection, and the PC can optionally have a minimum network status threshold that meets the screen projection conditions. If the PC determines that the current network status is lower than this minimum network status threshold, it is considered that the current network status cannot meet the screen projection requirements, resulting in a screen projection failure.

[0094] When the PC determines that the current network status does not meet the screen projection requirements, the screen projection fails and prompt message 3 is displayed. This prompt message 3 can remind the user to wait until the network is stable before trying to trigger the source device to cast the screen. For example, the user is prompted with "Network is unstable, please try again" to prompt the user to adjust and optimize the network status.

[0095] It is understandable that the above three scenarios are only examples and are not limitations on the embodiments of the present application. The PC's judgment on whether the corresponding application is protected by an application lock, whether the screen projection channel is occupied, and whether the network status meets the screen projection requirements are not required operations in the embodiments of the present application, and the judgment order is not limited. In some embodiments, please refer to Figure 7, which is a flow chart of a possible judgment mechanism. In response to the user clicking on the message card, the PC judges in each judgment node whether the application lock is set, whether the screen projection channel is occupied, and whether the network status meets the screen projection requirements. For example, as long as there is a judgment result that triggers the above-mentioned scenario of failure to start screen projection, this round of judgment terminates. After the user adjusts according to the prompts, he can click on the message card again to trigger a new round of judgment. Of course, in some embodiments, the judgment order of the judgment mechanism can be adjusted, and after a judgment node triggers a failure to start screen projection, the judgment process can stay at the judgment node and continue to detect whether the conditions for entering the next judgment node are met. The user adjusts according to the prompts, and when the conditions for entering the next judgment node are met, the judgment process automatically enters the next judgment node.

[0096] It can be understood that Figure 7 illustrates that after the PC detects that the user has selected a message card, the PC can first check whether the corresponding application is protected by an application lock (which can be called application lock detection), then check whether the communication channel used for screen projection is occupied (which can be called channel detection), and finally check whether the network conditions meet the requirements (referred to as network detection). In fact, in the embodiments of the present application, the order of the above-mentioned multiple tests is not limited. That is, in some embodiments, the PC can also first check whether the communication channel used for screen projection is occupied, then check whether the corresponding application is protected by an application lock, and finally check the network conditions.

[0097] In other possible embodiments, the PC may also simultaneously perform application lock detection, channel detection, and network detection, and display a corresponding reminder message when the test results indicate that at least one of the detection items has failed to trigger screen projection. For example, if the test results indicate that the application is protected by application lock and the communication channel is occupied, the PC will simultaneously display reminder message 1 and reminder message 2.

[0098] Of course, if the test results indicate that none of the above test items trigger screen projection failure, the PC determines that the mobile phone has the conditions for screen projection. In this way, as shown in Figure 8, the PC can display a screen projection window 201, which is used to display the application interface of the social application in the mobile phone. In the case of screen projection failure, the PC can display a corresponding reminder message.

[0099] In other embodiments, the source device can enable a distributed authentication function. That is, when the source device (e.g., a mobile phone) enables the distributed authentication function and is configured with an unlock password, if the target device (PC) passes the aforementioned tests (e.g., application lock test, communication channel test, and network test) and determines that the source device has the conditions for screen projection, then the target device can display the screen projection window, which is used to display the unlock verification interface. The unlock password can be an authentication password used to unlock the phone screen.

[0100] For example, after the PC detects that the user has selected the message card 104, if it determines that the mobile phone has the conditions for screen projection, the PC can display the screen projection window 201. The above-mentioned screen projection window 201 can first display the unlock verification interface. In this way, after the PC receives the mobile phone unlock password entered by the user and the unlock password is verified, the screen projection window 201 is controlled to switch to display "prompt waiting information". During the display of "prompt waiting information", the mobile phone responds to the instructions of the PC, starts the social application, and draws the application interface of the social application in the virtual interface, and obtains the drawing parameters of each display object in the virtual interface. Among them, the above-mentioned virtual interface is a service for the PC, which is not visible on the display screen of the mobile phone. The above-mentioned drawing parameters are related parameters for rendering the interface, which can also be called display information in some examples.

[0101] After the mobile phone completes drawing the above application interface, it sends the corresponding display information to the PC, instructing the PC to render the above application interface in the projection window 201. In this way, the projection window 201 switches to display the application interface of the social application (e.g., the application interface containing the application message corresponding to the message card 104). That is, the mobile phone projects the virtual interface to the PC's projection window 201. In other words, after the PC receives the mobile phone unlock password entered by the user and the unlock password is verified, the mobile phone starts to cast the screen to the PC.

[0102] As an implementation method, when a source device is casting its screen to a target device for the first time, if the source device has not set an unlock password, the target device does not need to display the unlock verification interface in the casting window. Instead, it instructs the source device to directly cast the screen to the target device. In this scenario, the target device can directly display a "waiting message" prompt. After the source device completes drawing the application interface to be cast, the target device controls the casting window to display the application interface to be cast.

[0103] When a source device projects its screen to a target device for the first time, if the source device has an unlock password set, that is, the user is required to perform password authentication before directly using the source device. In this case, after the target device determines that the source device is qualified for projection, it can trigger distributed authentication. For a description of distributed authentication, please refer to the above article and will not be repeated here.

[0104] In addition, it should be noted that when the source device performs heterogeneous screen projection to the target device for the first time and an unlock password is set, then regardless of whether the source device is in the locked screen state or unlocked state, the distributed authentication mechanism will be triggered when the target device instructs the source device to perform heterogeneous screen projection.

[0105] It should be noted that the first time the source device starts screen casting on the target device may refer to the first time the source device starts heterogeneous screen casting on the same target device, or it may refer to the first time the heterogeneous screen casting is started during the period from when the target device is turned on to when it is turned off. Specifically for the second explanation, assuming that the target device is turned on for the first time, and the target device is turned off for the first time after being used for a period of time, the first period from the first power-on to the first power-off is the first period of use. The target device is turned on for the second time, and the target device is turned off for the second time after being used for a period of time, and the second period from the second power-on to the second power-off is the second period of use. During the first period of use, the first time the source device starts heterogeneous screen casting on the target device belongs to the "first" start-up referred to in the second explanation; during the second period of use, the first time the source device starts heterogeneous screen casting on the target device also belongs to the "first" start-up referred to in the second explanation.

[0106] As an implementation method, in a scenario where this is not the first time that the source device is projecting screen to the target device, that is, the source device has previously initiated heterogeneous screen projection on the same target device, or the source device has initiated heterogeneous screen projection on the target device during the period from when the target device was powered on to when it was powered off.

[0107] Specifically, the second explanation is that during the first use period, the source device starts heterogeneous screen projection on the target device for the first time, and when the source device starts heterogeneous screen projection on the target device for the second time, the target device will determine that the source device has started heterogeneous screen projection on the target device; the same applies to the third, fourth, and other times thereafter. During the second use period, the source device starts heterogeneous screen projection on the target device for the first time. As can be seen from the above, the target device will determine that the source device has started heterogeneous screen projection on the target device for the first time; when the source device starts heterogeneous screen projection on the target device for the second time, the target device will determine that the source device has started heterogeneous screen projection on the target device; the same applies to the third, fourth, and other times thereafter. If the source device has started heterogeneous screen projection on the target device, in order to reduce the number of times the user enters the password while ensuring the privacy of the user, the target device can optionally first determine whether the time since the last distributed authentication exceeds the preset time threshold.

[0108] For example, when the time since the last distributed authentication does not exceed the preset time threshold, there is no need to re-authenticate the security. It should be noted that the preset time threshold can be a default value or a value set by the user, which indicates that within this time range, the screen projection connection between the source device and the target device still meets the requirements of protecting user privacy and security. Therefore, when the target device determines that the time since the last distributed authentication does not exceed the preset time threshold, the screen projection is started directly without displaying the unlock verification interface. This can avoid repeatedly requiring the user to enter the password, which causes cumbersome operations. It improves the convenience of users using heterogeneous screen projection while protecting user privacy and security.

[0109] As another example, when the time since the last security authentication is passed exceeds the preset time threshold, distributed authentication needs to be re-performed. It can be understood that when the target device determines that the time since the last distributed authentication is passed exceeds the preset time threshold, it is considered that the screen projection connection between the source device and the target device at this time no longer meets the requirements of protecting user privacy and security. The distributed authentication process in this case is the same as the distributed authentication process of "the source device starts heterogeneous screen projection on the target device for the first time" mentioned above, and will not be repeated here.

[0110] For example, assuming that the source device has set an unlock password and has not previously started heterogeneous screen projection on the target device. When the target device determines that the user has instructed the source device to project the screen, and that the source device has the conditions for projecting the screen, it can trigger a distributed authentication mechanism. After passing the distributed authentication, the first heterogeneous screen projection is successfully started; the first heterogeneous screen projection ends at time T1; assuming that the preset time threshold set by the user is 1 hour, at time T2 the target device detects again that the user has instructed the same source device to perform heterogeneous screen projection; the target device determines that the source device has started heterogeneous screen projection on the target device, and then determines whether the time from T2 to T1 is greater than 1 hour; the target device determines that the value of (T2-T1) is less than 1 hour, and directly starts the second heterogeneous screen projection.

[0111] In addition, after the mobile phone completes the heterogeneous screen projection to the PC, as shown in Figure 8, the mobile phone displays the main interface 101 and the screen projection window 201 on the PC. Among them, the screen projection window 201 displays the application interface of the screen projection application, such as the application interface 202 of the social application.

[0112] When the mobile phone receives a user instruction to turn off the screen, or when the mobile phone automatically turns off the screen without receiving any user operation for a long time, the display screen of the mobile phone enters a pseudo-screen-off state, and the PC's screen projection window 201 continues to display the application interface 202.

[0113] When the mobile phone receives a user-instructed lock screen operation, or when the mobile phone automatically locks the screen without receiving any user operation for a long time, the mobile phone displays the lock screen interface, and the PC's projection window 201 continues to display the application interface 202.

[0114] The following describes a system structure of an electronic device provided by an embodiment of the present application in conjunction with FIG9 .

[0115] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. It should be noted that the right half of Figure 9 shows the software architecture of a source device, such as a mobile phone. The left half of Figure 9 shows the software architecture of a target device, such as a tablet or PC.

[0116] In some embodiments, as shown in the right half of FIG9 , the software framework of the source device may include an application layer, a capability service layer, an application framework layer (framework, FWK), an Android runtime (Android runtime), a system library, and a driver layer.

[0117] (1) Application layer

[0118] The application layer may include a series of application packages, such as social, information, call, camera and other applications (also referred to as applications). For example, social applications include WeChat, QQ, etc., and information applications include text messages, etc. For the convenience of description, the application will be referred to as application below. The application on the source device can be a native application (such as an application installed in the source device when the operating system is installed before the source device leaves the factory), or it can be a third-party application (such as an application downloaded and installed by the user through the application store), which is not limited in the embodiments of this application.

[0119] (2) Capability service layer

[0120] The capability service layer provides capability support for implementing corresponding services. As shown in Figure 9, it may include a screen projection assistant module, a first virtualization service module, a first screen projection service module, a first device discovery authentication connection module, etc.

[0121] Among them, the screen projection assistant module can be a module used to interact with other electronic devices (such as target devices) to exchange screen projection related information, realize end-to-end logical control and connection management of screen projection, and generate a session key (SessionKey) to ensure the security of the communication session between the source device and the target device. The screen projection assistant module may include functional modules such as a screen projection management module, a heterogeneous switching module, and a virtualization service initialization module. The screen projection management module is responsible for managing screen projection related matters, such as adding and removing related processing logic settings of VirtualDisplay. Among them, VirtualDisplay can also be called a virtual display area, virtual screen or virtual Display, etc. The heterogeneous switching module is used to control the source device to switch from a heterogeneous screen projection mode to a homogeneous screen projection mode. The virtualization service initialization module makes initialization settings for the virtualization service that starts heterogeneous screen projection.

[0122] The first virtualization service module is used to implement end-to-end logical control of audio and video stream projection, and is responsible for the audio and video data streams generated by speakers, microphones, cameras, etc. transmitted between the source device and the target device.

[0123] The first screen projection service module may include a first display virtualization module and a first reverse control module, which respectively provide screen projection capabilities (i.e., the content of the source device is projected to the target device for display) and reverse events (i.e., events that control the source device triggered by the target device) basic control capabilities. The first screen projection service module receives instructions from the screen projection management module and provides corresponding screen projection services according to the instructions, thereby realizing the projection of programs running on the local machine to other electronic devices (such as the target device). The first screen projection service module injects reverse control and input method into the display of the target device (hereinafter referred to as the external screen), and adds a virtual display tag (such as Flag1) when creating a virtual display area, so that the external screen only displays its own content, and does not mirror the content of the display of the source device (hereinafter referred to as the main screen), thereby realizing heterogeneous screen projection.

[0124] The first device discovery connection authentication module is responsible for discovery, connection, authentication and other tasks, and is used to control the driver layer to implement proximity discovery, authentication, connection and other functions between the source device and the target device, and may include a first communication module and a first security authentication module. The first communication module provides a signaling transmission channel between the source device and the target device, and is used to send instructions to the driver layer to establish a communication connection and a screen projection connection between the source device and the target device. At the same time, the first communication module is also used to start the screen projection assistant module after the transmission channel is established. The first security authentication module is used for security authentication, and is responsible for identifying the owner and starting distributed unlocking authentication, obtaining a trust ring, and ensuring the privacy and security of the user.

[0125] (3) Application Framework Layer (FWK)

[0126] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. As shown in Figure 9, the application framework layer can include the Activity Manager Service (AMS), the Window Manager Service (WMS), the multi-screen framework module, the System UI module, and so on.

[0127] The AMS is responsible for managing activities, starting, switching, and scheduling various components in the system, as well as managing and scheduling applications. For each activity, AMS has a corresponding application record (ActivityRecord), which records the status of the application's activities. Each activity creates a corresponding ActivityRecord object in AMS, which AMS uses as an identifier to schedule the application's activity processes. These ActivityRecord objects are managed in their respective task stacks, and each task stack can have multiple ActivityRecord objects. All task stacks are centrally managed by the ActivityStack, which is responsible for the order, push, and pop of TaskStacks. Specifically, AMS defines data classes for storing processes, activities, and tasks. The data class corresponding to a process can include process file information, memory status information, and the activities and services contained in the process. Activity information can be stored in the ActivityStack, which is used to centrally schedule application activities. The ActivityStack specifically stores information about all currently running Activities (i.e., the final ArrayList mHistory), such as interface configuration information. For example, the currently running Activity can be stored in the newArrayList. The ActivityStack also stores information about previously run Activities, such as interface configuration information. It's important to note that an Activity does not correspond to a single application; only an ActivityThread corresponds to an application. Therefore, Android allows multiple applications to run simultaneously, which actually means it allows multiple ActivityThreads to run simultaneously. The basic concept of activity scheduling in Android is this: each application process reports to the Active Directory Management Service (AMS) when it wants to start a new Activity or stop the current Activity. The AMS internally keeps a record of all application processes. When the AMS receives a start or stop report, it first updates the internal record and then notifies the corresponding client process to run or stop the specified Activity. Because the AMS maintains a record of all Activities, it can schedule these Activities and automatically close background Activities based on the status of the Activities and system memory.

[0128] WMS carries data and attributes related to the "interface" and is used to manage and control the status related to the "interface", that is, to manage the graphical user interface (GUI) resources used on the mobile phone screen, such as managing window programs and event dispatching. Among them, managing window programs refers to outputting to the physical screen or other display devices in an orderly manner with the assistance of the application server and WMS according to the display request of the application. Event dispatching refers to dispatching user events from the keyboard, physical buttons, touch screen, mouse, trackball (TraceBoll), etc. to the corresponding controls or windows. The window management program can also obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc., and is responsible for managing the display mode of the application window, including the coordinate size of the window display, the window display level, etc. Specifically, it may include: creating and destroying windows, showing and hiding windows, window layout, focus management, and input method and wallpaper management, etc. In some embodiments of the present application, WMS can ensure the visibility of the projection window when the mobile phone is in the locked screen and screen off state.

[0129] It should be noted that the task stack management function of AMS is used to manage the life cycle of Activities and the scheduling between Activities. The life cycle management function of WMS is used to manage the life cycle of windows. The two can cooperate with each other to activate the window through the callback (onResume) interface, and the activated window is displayed in the foreground; or, the window can be paused through the pause (onPause) interface to switch the window from the foreground to the background. Taking the topmost window (top Activity) in the task stack as an example, the top Activity calls onResume to activate it, then the top Activity is displayed in the foreground. When a new window is pushed into the task stack, and the new window is located on the upper layer of the original top Activity, the original top Activity calls onPause to pause the window to switch the original top Activity from the foreground to the background. A similar principle can be used to manage the life cycle of the new window.

[0130] The multi-screen framework module is responsible for implementing the heterogeneous screen projection logic, calling the PendingIntetRecord interface to start the Activity corresponding to the PendingIntet (specifying the send behavior), and launching the application corresponding to the Activity into the virtual display area by adding a heterogeneous screen projection tag (e.g., Flag2). PendingIntent is a capability provided by Android for external programs to call their own programs. Its lifecycle is independent of the main program, and it executes specified behaviors when certain conditions are met or certain events are triggered. The multi-screen framework module is also used to monitor sensor events and interact with the screen projection assistant module. For example, it calls ActivityRecord to monitor whether the Activity is in the foreground and visible, and notifies the screen projection assistant module of the monitoring results; it calls Task to monitor whether the virtual display area stack is empty and notifies the screen projection assistant module of the monitoring results. The multi-screen framework module can be used to translate and encapsulate the original input events, obtain input events containing more information, and send them to the WMS, which stores the clickable areas (such as controls) of each application and the location information of the focus window. Therefore, the WMS can correctly distribute input events to the specified control or focus window. The multi-screen framework module can respond to the received input events and forward the input events to the WMS. The WMS distributes them to the corresponding applications. The applications send registration requests to the AMS and the applications are started.

[0131] In addition, the multi-screen framework module is also used to switch the application projected to the external screen back to the main screen when the user clicks the recent task card or desktop icon. At this time, if the application projected to the external screen adopts a heterogeneous screen projection method, when switching the application back to the main screen, it is necessary to call the heterogeneous switching module mentioned above to switch the heterogeneous screen projection method to the homogeneous screen projection method, otherwise the application cannot be displayed on the main screen. The multi-screen framework module is also used to remove the security layer when the source device is in the locked screen state. In some embodiments, the multi-screen framework module can also be used to make relevant modifications to the power button of the mobile phone in the screen projection scenario.

[0132] The SystemUI module is a set of UI components that provide users with system-level information display and interaction. It is mainly used to implement status bar information display (such as battery, Wi-Fi signal, 3G / 4G icon display), notification panel (such as system messages and third-party message notifications), recent taskbar display panel (such as displaying recently used applications), etc. In the embodiment of the present application, the SystemUI module is responsible for notification services, which is used to transfer message notifications from the source device to the target device.

[0133] It is understood that the application framework layer may also include a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. (not shown in FIG9 ). The specific meanings thereof may be referred to in relevant technical documents and will not be described here.

[0134] (4) Android runtime and system libraries

[0135] The Android runtime is responsible for system scheduling and management. It consists of core libraries and a virtual machine. The core libraries consist of two parts: one containing the functional functions called by programming languages ​​(e.g., Java) and the other containing the system's core libraries. The application layer and application framework layer run in the virtual machine. The virtual machine executes the application layer and application framework layer programming files (e.g., Java files) as binary files. The virtual machine performs functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0136] The system library can include multiple functional modules. For example: Surface Manager, Media Libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc. The surface manager is used to manage the display subsystem and provides the fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications. The media library supports the playback and recording of various common audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0137] (5) Driver layer

[0138] The driver layer is the foundation of the Android operating system, and all of the Android operating system's ultimate functions are completed through the driver layer. As shown in Figure 9, the driver layer includes low-level drivers, responsible for tasks such as discovery, authentication, and connection. For example, the driver layer receives commands from the communication module in the application framework layer and executes actions such as connection and disconnection. Specifically, the driver layer includes Wi-Fi P2P chips and Bluetooth chips, which can establish Wi-Fi P2P and Bluetooth transmission channels. Of course, the driver layer can also include hardware drivers such as display drivers, camera drivers, audio drivers, sensor drivers, and virtual card drivers.

[0139] It should be noted that the software structure diagram of the electronic device shown in Figure 9 provided in this application is only an example and does not limit the specific module divisions in different layers of the Android operating system. For details, please refer to the introduction of the Android operating system software structure in conventional technology. In addition, the screen projection method and device provided in this application can also be implemented based on other operating systems, and this application will not give examples one by one.

[0140] In some embodiments, as shown in the left half of Figure 9, the software framework of the target device may include a PC manager, an operating system, and a driver layer. Some functional modules of the target device and their functions can refer to the above-mentioned source device and will not be repeated here.

[0141] Specifically, the PC manager of the target device includes a message center module, a second virtualization service module, a second screen projection service module, and a second device discovery connection authentication module.

[0142] The Message Center module is used to interact with messages and signaling from the source device, receiving message notifications from the source device and initiating session instructions (openSession) to the source device. It is also used to issue instructions to start the screen projection service after a session is established.

[0143] The functions of the second virtualization service module, the second screen projection service module, and the second device discovery authentication connection module can respectively refer to the first virtualization service module, the first screen projection service module, and the first device discovery connection authentication module in the above-mentioned source device. The difference is that the second screen projection service module is used to receive and parse video streams and related information sent by other devices, so as to display applications running on other devices on the local device.

[0144] The operating system of the target device can be Windows OS (such as a personal computer) or Android OS (such as a tablet).

[0145] For the driver layer of the target device, please refer to the above source device.

[0146] Based on the system structure of the electronic device shown in Figure 9, taking a mobile phone as the source device (such as called the first device) and a PC as the target device (such as called the second device) as an example, combined with Figures 10 to 12, the process of projecting the screen of the aforementioned mobile phone to the PC when the screen is on and unlocked is explained.

[0147] For example, the mobile phone side includes a first communication module, a screen projection assistant module, a first virtualization service module, a first screen projection service module, a first security authentication module, FWK (including a SystemUI module), and an application; the PC side includes a message center module, a second screen projection module, a second communication module, and a second security authentication module. It should be noted that the other functional modules shown in Figure 9 also participate in the specific implementation process of a screen projection method provided in an embodiment of the present application, and are not shown in this timing diagram.

[0148] As shown in FIG10 , the screen projection method may include the following steps:

[0149] S101: The message center module of the PC receives an operation of selecting a mobile phone by a user.

[0150] It is understandable that S101 is only an example of establishing a first communication connection between a PC and a mobile phone. The PC detects a device in the same local area network as the PC, displays all connectable devices for the user to select, and the user selects the mobile phone on the PC to initiate subsequent connection behavior. There are other ways to initiate the first communication connection between the PC and the mobile phone. For example, the mobile phone detects a device in the same local area network as the PC, displays all connectable devices for the user to select, and the user selects the PC on the mobile phone to initiate subsequent connection behavior. For another example, the mobile phone and the PC can also establish a near-field connection through the "touch and pay" function, the NFC function, etc., which is not limited here.

[0151] S102: The message center module sends a message to the second communication module instructing to establish a communication connection with the mobile phone. The message carries the device identification code of the mobile phone.

[0152] The user selects a mobile phone as a collaborative notification device from among the available collaborative notification devices found by the PC. After receiving the selection operation input by the user, the message center module of the PC can send an inter-process communication (IPC) message to the second communication module. Optionally, the IPC message can carry the device name, device identification code, MAC address, etc. of the mobile phone, which is used to instruct the second communication module to establish a communication connection with the mobile phone.

[0153] S103: The second communication module establishes a first communication connection with the first communication module of the mobile phone.

[0154] Exemplarily, the first communication connection may be a Bluetooth connection (BR connection), which is used to implement collaborative notifications between the PC and the mobile phone, that is, messages on the mobile phone can be transferred to the PC through the Bluetooth channel.

[0155] S104: The second communication module returns a message to the message center module indicating that the first communication connection is successfully established.

[0156] S105: The application of the mobile phone receives an application message.

[0157] After the PC and the mobile phone establish the first communication connection, the mobile phone receives an application message pushed by the application installed on the mobile phone. The application message can have different forms of expression in the application on the mobile phone, the SystemUI module, and the message center module on the PC, collectively referred to here as application messages.

[0158] S106: The application sends an application message to the SystemUI module. The application message carries the message content and the application package name.

[0159] S107: The SystemUI module instructs the first communication module to forward the application message to the PC.

[0160] For example, in an embodiment of the present application, after the PC and the mobile phone establish a first communication connection, the SystemUI module will transfer all messages received by the mobile phone to the PC based on the first communication connection; it can also be transferred selectively, for example, messages with a special tag will be transferred to the PC, or messages from specific applications will be transferred to the PC, or messages judged to be of higher importance will be transferred to the PC, etc., which are not limited here.

[0161] S108: The first communication module sends an application message to the second communication module based on the first communication connection. The application message carries message content and an application package name.

[0162] S109: The second communication module sends an application message to the message center module. The application message carries the message content and the application package name.

[0163] S110: The message center module receives the application message and controls the display screen to display the corresponding message notification control.

[0164] After the PC and mobile phone establish a first communication connection, application messages received on the mobile phone are transferred to the PC. It is understood that whether the mobile phone is unlocked and the screen is on, locked, or off does not affect the flow of message notifications between the PC and the mobile phone. After the PC receives the application message, it displays the message notification control associated with the application message.

[0165] In some embodiments, the target device receives a user operation on the message notification control, the target device establishes a third communication connection with the source device, the target device sends a request to start heterogeneous screen projection to the source device, and the source device responds to the request by displaying the application projection window on the target device. That is, as shown in Figures 11 and 12, the above method may further include:

[0166] S201: The message center module receives a first operation of a user clicking a message notification control.

[0167] Among them, the message notification control can be called a first control, and the first control displays an application message of the first application, such as called a first message.

[0168] S202: The message center module sends a message to the second communication module instructing to create a second session. The message carries the service channel name (WI-FI P2P channel) and the device identification code of the mobile phone.

[0169] Among them, exemplarily, the service channel name can be a WI-FI P2P channel, the device identification code is the identification code of the mobile phone, and the above-mentioned instruction to create a second session is used to instruct the second communication module to establish a WI-FI P2P channel between the PC and the mobile phone corresponding to the device identification code.

[0170] S203: The second communication module establishes a third communication connection with the first communication module.

[0171] Exemplarily, the third communication connection may be a WI-FI P2P connection, providing a data transmission channel for the screen projection service.

[0172] It should be noted that the types of the first communication connection, the second communication connection, and the third communication connection established between the PC and the mobile phone are not limited to Bluetooth connection and Wi-Fi P2P connection, and can also be other types of connection methods as long as they can meet the requirements for data transmission between the two.

[0173] S204: The first communication module sends a message instructing the screen projection assistant module to start.

[0174] S205: The screen projection assistant module is started.

[0175] The first communication module is also responsible for interacting with the screen projection assistant. The screen projection assistant module is in a closed state when the mobile phone has no screen projection demand. The first communication module triggers the screen projection assistant module to start when there is a screen projection demand.

[0176] S206: The second communication module returns a message to the message center module indicating that the third communication connection is successfully established.

[0177] S207: Based on the third communication connection, the message center module negotiates parameters with the screen projection assistant to determine the session key and the IP address of the other party.

[0178] Parameter negotiation involves negotiating a SessionKey on the service side and obtaining the peer device's IP address. The SessionKey, also known as a data encryption key or working key, is a randomly generated encryption and decryption key used to ensure secure communication between a user and another computer, or between two computers. It is negotiated between communicating users and is typically generated dynamically, only when session data encryption is required.

[0179] S208: The message center module sends a message indicating startup to the second screen projection service module, which carries the session key and the IP address of the mobile phone.

[0180] S209: The second screen projection service module is started and initialized according to the session key and the IP address of the mobile phone.

[0181] S210: The second screen projection service module returns a message indicating that initialization is completed to the message center module.

[0182] As can be seen from the above, the screen projection service module provides screen projection capabilities and basic reverse event control capabilities. After the screen projection service on the PC side is successfully initialized, it provides the ability to receive and parse the relevant information sent by the mobile phone side, so that the application running on the mobile phone can be displayed on the PC side. At the same time, the basic reverse event control capability provides support for heterogeneous screen projection. Therefore, before performing heterogeneous screen projection services, the screen projection service module needs to be initialized to ensure the normal provision of screen projection capabilities and basic reverse event control capabilities.

[0183] S211: The message center module sends a message to the second screen projection service module instructing to start the screen projection service.

[0184] S212: Based on the third communication connection, the second screen projection service module sends a message to the screen projection assistant module to start the screen projection service.

[0185] It should be noted that step S207 includes the second screen projection service module sending a message to the second communication module, the second communication module then transmits the message to the first communication module through the third communication connection, and the first communication module then sends the message to the screen projection assistant module. For the convenience of description, the following "based on the third communication connection" includes the data transmission process between the first communication module and the second communication module, which will not be repeated below. In addition, the above-mentioned message for starting the screen projection service can also be called the first request. When the user selects the first control to display the first message, the first request can carry the first identifier.

[0186] S213: The screen projection assistant module is bound to the screen projection service.

[0187] It can be understood that the screen projection management module in the screen projection assistant module is bound to the relevant screen projection services to manage screen projection affairs.

[0188] S214: The screen projection assistant module sends an initialization instruction message to the first virtualization service module.

[0189] It can be understood that the virtualization service initialization module in the screen projection assistant module is used to instruct the virtualization service module to initialize.

[0190] S215: The first virtualization service module completes initialization.

[0191] The first virtualization service module is responsible for the audio and video virtualization in the screen projection service. The interaction between the first virtualization service module on the mobile phone and the second virtualization service module on the PC can enable the mobile phone application projected to the PC to use the PC's speaker, microphone, and camera during the screen projection process, and transmit the generated audio and video data streams between the mobile phone and the PC, thereby realizing functions such as making and receiving audio and video calls during screen projection.

[0192] S216: The first virtualization service module sends a message to the first screen projection service module to instruct the start of the screen projection service.

[0193] S217: The first screen projection service module is started.

[0194] S218: The first screen projection service module creates a virtual display and adds a virtual display tag to it.

[0195] It should be noted that the screen projection service module is an important functional module for implementing heterogeneous screen projection. The specific implementation method is that the first screen projection service module on the mobile phone adds Flag1: VIRTUAL_DISPLAY_FLAG_OWN_CONTENT_ONLY (virtual display tag, also known as the second tag) when creating the virtual display area. It is used to realize that the screen projection window on the PC only displays its own content, and does not mirror the content displayed on the mobile phone display, thereby ensuring that the display content of the mobile phone does not affect the display content of the PC screen projection window.

[0196] In this way, when the mobile phone is in use, the mobile phone displays other application interfaces or desktops, which will not affect the display of the PC's projection window. The PC's projection window can maintain its own content without being synchronized with the display content of the mobile phone. At the same time, the basic reverse event control capabilities provided by the projection service module can achieve reverse control of the mobile phone on the computer side. Of course, the implementation of the heterogeneous projection method also requires the second projection service module on the PC side to interact with the first projection service module on the mobile phone side.

[0197] S219: The first screen projection service module sends a message to the screen projection assistant module indicating that the screen projection service has been started.

[0198] S220: Based on the third communication connection, the first screen projection assistant sends a message to the message center module indicating that the screen projection service has been started.

[0199] S221: Based on the third communication connection, the message center module sends a message to FWK to start the application for screen projection, carrying the application package name.

[0200] The application package name carried is the package name of the application (e.g., a social application) corresponding to the message notification control (e.g., message card 104) selected by the user in S201. It is understood that, as shown in FIG1 , message card 104 is used to display application messages from the social application in the mobile phone, and the message card 104 can be seen as corresponding to the social application in the mobile phone. The corresponding relationship between other message notification controls and applications is similar.

[0201] S222: FWK starts the application screen projection window and marks the screen projection tag for all activities corresponding to the application package name.

[0202] In some embodiments, the process of FWK starting the application projection window can refer to the relevant technology, and will start the application corresponding to the application package name, and draw the application interface of the application in the virtual display area (first display area), and generate display data, which will not be described in detail. Different from the relevant technology, in the process of starting the application projection window, FWK can mark the projection label (first label) for all activities corresponding to the above application package name (that is, the application package name carried in the message of starting the application for projection). All activities corresponding to the above application package name can be all activities corresponding to the application indicated by the application package name. For example, as shown in Figures 1 and 2, since the user selects the message card 104 on the PC, the message sent by the PC to the mobile phone to start the application for projection will carry the application package name of the social application. In this way, when the mobile phone executes S222, it is necessary to mark all activities of the social application with the projection label. In this way, through the projection label, it is represented that the application indicated by the application package name is performing heterogeneous projection.

[0203] S223: FWK sends the display data of the application screen projection to the first screen projection service module.

[0204] S224: Based on the third communication connection, the first screen projection service module sends the display data of the application projection to the second screen projection service module.

[0205] S225: The second screen projection service module controls the display screen to render the above display data to the screen projection window.

[0206] The screen projection window may also be referred to as the first window, and the display data rendered to the first window may be image information. For example, when the mobile phone projects the screen of a first application, the display data rendered in the first window may be the first image information corresponding to the first application. In the subsequent embodiments, the screen projection of the first application is used as an example for description.

[0207] S226: The second screen projection service module sends a message to the message center indicating that the application screen projection interface of the projection window has been rendered.

[0208] S227: The message center module controls the display screen to display the application projection window.

[0209] At this point, the PC displays the application projection window corresponding to the application message, that is, the phone successfully projects its screen to the PC when the screen is on and unlocked.

[0210] For example, as shown in Figures 1 and 2, when the PC receives the user's click on the message card 104, the PC will display the projection window 201 corresponding to the social application, which is used to display the application interface 202 corresponding to the application message. The above application projection window can be the projection window 201 for displaying the application interface 202.

[0211] After successfully projecting the phone's screen to the PC, that is, after S227, the phone may enter the screen-off or lock state. For example, the user may instruct the phone to enter the screen-off or lock state using the phone's lock button. Another example is the phone automatically entering the screen-off or lock state after receiving no user input for a long period of time.

[0212] The following describes, in conjunction with Figure 13, the process of interaction between the various functional modules within the mobile phone when the mobile phone goes from the screen-on, unlocked state to the screen-off or locked state during screen projection.

[0213] Exemplarily, the mobile phone may also include PowerManagerService, KeyguardServiceDelegate (lock screen management service), MultiDisplayPowerManager (multi-screen framework module), ActivityTaskManagerService (activity stack management service), KeyguardController (lock screen controller), ActivityRecord (activity recorder), Task (task stack), RootWindowsContainer (root window container), WindowsCastManager (window appearance manager), and SurfaceFlinger (layer compositor).

[0214] As shown in FIG13 , the above method may include:

[0215] S301, PowerManagerService receives an operation instructing to lock the screen.

[0216] Among them, PowerManagerService is responsible for the power management of the mobile phone system. Common functions include: turning on the screen, turning off the screen, entering the screen saver, etc. PowerManagerService inherits from SystemService, is started by SystemServer, registered in the system service, and interacts with other components through Binder.

[0217] In some embodiments, after the PowerManagerService receives a press operation from the user via the lock screen key (ie, the Power key) of the mobile phone, it can determine that an operation instructing to lock the screen has been received.

[0218] S302, PowerManagerService sends a lock screen instruction to KeyguardServiceDelegate.

[0219] In some embodiments, the KeyguardServiceDelegate is a class for proxying the KeyguardService. The PowerManagerService sends a lock screen instruction to the KeyguardServiceDelegate to instruct the startup of the lock screen loading process.

[0220] S303, KeyguardServiceDelegate sends power management information to MultiDisplayPowerManager.

[0221] In some embodiments, when KeyguardServiceDelegate determines that a screen projection service is in progress between the mobile phone and the PC, it instructs MultiDisplayPowerManager to take over PowerManagerService to manage the power of the mobile phone system through power management information.

[0222] Of course, in addition to receiving a user-instructed screen lock operation, the phone can also trigger the MultiDisplayPowerManager to take over the PowerManagerService to manage the phone system power when the screen is automatically turned off or the phone receives a user-instructed screen turn off operation. In other words, in some possible scenarios, the above method may not include S301-S302, but include the following S304-S306.

[0223] S304, PowerManagerService determines that the display screen is automatically turned off.

[0224] In some embodiments, the process of determining whether the display screen automatically turns off can refer to relevant technologies and will not be repeated here.

[0225] S305, PowerManagerService detects the projection tag of the virtual display screen and interrupts the screen off processing for the virtual display screen.

[0226] In some embodiments, if PowerManagerService detects the projection tag of the virtual display, it indicates that the phone is currently projecting its screen to another device and that projection is in progress. In this scenario, to prevent the automatic screen off from affecting the projection effect, PowerManagerService can interrupt the screen off process for the virtual display.

[0227] S306: PowerManagerService sends power management information to MultiDisplayPowerManager.

[0228] In some embodiments, by sending the above power management information, MultiDisplayPowerManager can be instructed to take over PowerManagerService to manage the power of the mobile phone system.

[0229] It is understandable that the above steps S301-S303 and S304-S306 describe two different scenarios in which power management information is sent to the MultiDisplayPowerManager. When the phone is in the locked screen state, S301-S303 are executed; when the phone is in the off screen state, S304-S306 are executed. There is no correlation between S301-S303 and S304-S306.

[0230] S307, MultiDisplayPowerManager controls the display screen to enter a fake screen-off state.

[0231] It should be noted here that the above-mentioned fake screen-off state, screen-off state, and screen-on state are state types of the display screen. Among them, compared with the screen-on state and the screen-off state, not only are the online and offline states of the LCD of the display screen different (for example, the LCD is in the power-on state in the screen-on state, and the LCD is in the offline state in the screen-off state), but the configuration parameters corresponding to the internal related functional services (such as AOD service and POWER service) are also different. Compared with the screen-on state, the fake screen-off state is only different in the online and offline states of the LCD of the display screen (for example, the LCD is in the power-on state in the screen-on state, and the LCD is in the offline state in the fake screen-off state), but the configuration parameters corresponding to the internal related functional services (such as AOD service and POWER service) are the same. Compared with the screen-off state, the online and offline states of the LCD of the fake screen-off state and the screen-off state are the same, but the configuration parameters corresponding to the internal related functional services (such as AOD service and POWER service) are also different.

[0232] Usually, the screen-off state and the screen-on state are managed by PowerManagerService. In the embodiment of the present application, after MultiDisplayPowerManager takes over PowerManagerService for power management, the MultiDisplayPowerManager can control the display screen to enter a fake screen-off state.

[0233] In some embodiments, MultiDisplayPowerManager can respond to the above power management information and switch the display screen of the mobile phone from the bright screen state to the fake screen off state. During the switching process, MultiDisplayPowerManager only needs to control the LCD power off through the TP service without changing the status and configuration of related services.

[0234] After PowerManagerService instructs MultiDisplayPowerManager to take over power management, and MultiDisplayPowerManager completes controlling the display screen to switch from the on-screen state to the pseudo-off-screen state, the method may further include:

[0235] S308, PowerManagerService instructs ActivityTaskManagerService to configure the application projection window to be visible in the foreground.

[0236] S309, ActivityTaskManagerService sends a message to KeyguardController instructing to set the Activity with the projection tag to the OnResume state.

[0237] It should be noted that ActivityTaskSupervisor listens to whether the Activity is in the OnResume state by calling KeyguardController. If so, it means that the application projection window corresponding to the Activity is visible to the user and can interact with the user after it is launched on the PC (OnResume state); if the Activity is in the stopped state (OnPause state), it means that the application projection window corresponding to the Activity is in the background after it is launched on the PC, and the user cannot operate it. Among them, the OnResume state and the OnPause state are two states in the life cycle of the Activity. When the Activity is in the OnResume state, the Activity is in an activated state that is visible and can interact with the user, and the user can get the focus of the activity; when the Activity is in the OnPause state, the Activity is covered by another transparent or Dialog-style Activity. At this time, it still maintains a connection with the window manager, and the system continues to maintain its internal state. It is still visible, but it has lost focus and cannot interact with the user. When the phone is in the locked or off screen state, the AMS component will switch the Activity lifecycle to the onPause state during the process of starting the Activity, causing the Activity to be unable to interact with the user, resulting in screen projection exceptions.

[0238] S310, KeyguardController identifies the Activity with the projection tag and sets it to the OnResume state.

[0239] KeyguardController is a lock screen controller that identifies an Activity with the cast label and returns "true" to indicate that the Activity is set to the OnResume state, ensuring that when the source device is in the lock screen state, the application cast window corresponding to the Activity with the cast label is visible on the target device.

[0240] S311, KeyguardController sends a message to ActivityRecord instructing to keep the Activity with the projection tag in the OnResume state.

[0241] S312, ActivityRecord controls the Activity with the screen projection tag to maintain the OnResume state.

[0242] S313, ActivityRecord sends a verification instruction message to Task, instructing Task to verify the status of the Activity at the top of the task stack.

[0243] S314, Task determines that the Activity at the top of the task stack is in the OnResume state.

[0244] It's important to note that an application may correspond to one or more activities, which are stored in one or more task stacks. Task stacks follow a last-in, first-out order, with the activity at the top of the task stack being executed first. To ensure that the application projection window corresponding to the upcoming activity is visible, ensure that the activity at the top of the stack is in the OnResume state.

[0245] It can be understood that steps S313 and S314 are equivalent to a fault-tolerant mechanism, which checks whether the Activity at the top of the task stack is in the OnResume state, so these two steps are optional, that is, in other embodiments, S313 and S314 may not be included.

[0246] S315, Task instructs RootWindowsContainer to control all tasks to enter a dormant state.

[0247] S316, RootWindowsContainer traverses the activities of all tasks, skips the activities with the projection tag, and controls other activities to enter the dormant state.

[0248] It should be noted that except for the activity with the screen projection tag, which skips the hibernation process, all other activities will be put into hibernation. Therefore, RootWindowsContainer needs to traverse all activities corresponding to the task, filter out the activities with the screen projection tag, and return "false" to indicate that the activity will not enter hibernation; for other activities, it returns "true" to enter the hibernation process.

[0249] For example, Activity1 and Activity2 in the same task stack are both added with the heterogeneous screen projection tag, and RootWindowsContainer causes Activity1 and Activity2 to skip the sleep process; while Activity3 and Activity4 that are not added with the heterogeneous screen projection tag are put into sleep state.

[0250] When Activity1 is at the top of the task stack, Task checks the status of Activity1 to ensure that it is in the OnResume state. Then, after Activity1 is projected onto the screen from a different source, it is visible to the user on the PC and can be operated by the user in the foreground.

[0251] After executing Activity1, Activity2, which is not dormant at this time, progresses to the top of the task stack. Task verifies the status of Activity2 to ensure that it is in the OnResume state. Then, Activity2 is visible to the user on the PC after being projected from another source and can be operated by the user in the foreground. The execution process of other Activities that have been marked with projected from another source is similar and will not be repeated here. Activities that have not been marked with projected from another source will be put into dormant state by RootWindowsContainer and will not be started to be displayed on the external screen.

[0252] Through the above steps, the application screen projection window on the PC is not affected by the status of the mobile phone and can still maintain foreground visibility and operability. Then, as shown in Figure 13, the above method may further include:

[0253] S317, MultiDisplayPowerManager sends a cancel message to WindowsCastManager, where the cancel message indicates to cancel drawing of the lock screen layer in the virtual display area.

[0254] S318, WindowsCastManager determines not to draw the lock screen layer in the virtual display area.

[0255] S319, MultiDisplayPowerManager instructs SurfaceFlinger not to switch to same-source projection.

[0256] S320, SurfaceFlinger configures the virtual display area to not mirror display content in the visible display area.

[0257] SurfaceFlinger receives graphics display data from multiple sources, synthesizes them, and then sends them to the display device. For example, when you open an application, there are three common display layers: the status bar at the top, the navigation bar at the bottom or side, and the application interface. Each layer is updated and rendered separately. These interfaces are synthesized by SurfaceFlinger into a single application layer display data that is refreshed to the hardware display.

[0258] In an embodiment of the present application, SurfaceFlinger determines that the virtual display area has a virtual display tag, then when synthesizing the application layer, it only obtains the layer on the virtual display area, and does not obtain the layers of other display areas to synthesize with it, so that the aforementioned heterogeneous screen projection method can be achieved, that is, the synthesized application layer is only related to the layer of the virtual display area. Please refer back to Figure 2. SurfaceFlinger will not obtain the layers of the visible display area, so even if the visible display area includes the lock screen interface or other application interfaces, it will not affect the interface displayed by the PC's projection window. Because SurfaceFlinger only obtains the layers of the virtual display area and passes them to the PC's projection window for display in some way.

[0259] After SurfaceFlinger completes the synthesis of the display data, it can be sent through the first screen projection service module to instruct the PC to display the corresponding application projection window. For details, please refer to S223 to S227. In addition, when the phone is in the bright screen state, the application interface of the first application can be drawn with the virtual display area, and then the corresponding display data is synthesized by SurfaceFlinger, such as the first image information. After the phone enters the lock screen or the screen-off state, due to the processing of the aforementioned steps, the application interface of the first application can still be drawn with the virtual display area, and the corresponding display data is synthesized by SurfaceFlinger, such as the second image information. Since the above-mentioned first image information and second image information both correspond to the application interface of the first application, in some scenarios, the corresponding display content of the two may be the same.

[0260] It is understood that the application layer may include the application interface corresponding to the currently executed Activity. The display data may include information such as different display objects in the application layer, the display position and size of the display objects in the virtual display screen, etc.

[0261] It should be noted that the virtual display area is the virtual display area marked Flag1 created in step S218. The content of the virtual display area is not visible to the user, and the mobile phone can use the virtual display area as a canvas that occupies a certain storage space. For example, please refer to Figure 16, which is a process diagram for starting the first application projection window. When the application package name carried by the first request to start the projection corresponds to application 1, the PendingIntentRecord will mark all Activities of application 1 with a projection label, as shown in the task stack (created by Task) in Figure 16. When executing Activity1 of application 1 (Activity1 corresponds to the first message), as shown in Figure 16, the mobile phone draws the first application interface corresponding to Activity1 in the virtual display area. SurfaceFlinger determines that the virtual display area has a virtual display label, then only obtains the first application interface layer in the virtual display area, and generates the corresponding first display data and sends it to the target device.

[0262] After receiving the first display data, the target device renders the application projection interface corresponding to the first message of the first application based on the first display data and displays it in the projection window of the target device. For details, please refer to steps S225-S227 in Figure 12 and Figure 7. At this time, the projection window of the target device displays the application projection interface corresponding to the first display data.

[0263] When executing Activity2 of Application 1 (Activity2 is pushed to the top of the task stack), the phone draws the second application interface corresponding to Activity2 in the virtual display area. SurfaceFlinger determines that the virtual display area has a virtual display tag, then only obtains the second application interface layer in the virtual display area, and generates the corresponding second display data and sends it to the target device. The projection window of the target device then displays the application projection interface corresponding to the second display data, that is, the application projection interface corresponding to the first display data is switched to the application projection interface corresponding to the second display data, completing the switching of different application interfaces of Application 1 in the projection window.

[0264] For example, in some embodiments of the present application, please refer to Figure 17, which is a process diagram for starting the second application projection window. In the projection method provided in the embodiment of the present application, when the projection window of the target device displays the application projection interface corresponding to the first application (such as, the projection window of the target device displays the application projection interface corresponding to the second display data), the source device receives a second message from the second application (also called the second data). After the second message is transferred to the target device, the target device displays the message notification control corresponding to the second message. After the target device receives the operation of the source device clicking the message notification control, it sends a second request to the source device to start the projection. When the source device is in the locked screen or off screen state, the first projection service module receives the second request, creates a second virtual display area (VirtualDispaly2) and adds a virtual display tag. When executing Activity3 of Application 2 (Activity3 corresponds to the second message), as shown in Figure 17, the mobile phone draws the third application interface corresponding to Activity3 in the virtual display area. SurfaceFlinger determines that the virtual display area has a virtual display tag, then only obtains the third application interface layer in the virtual display area, and generates the corresponding third display data and sends it to the target device.

[0265] After receiving the third display data, the target device draws and renders the application projection interface corresponding to the second message of the second application according to the third display data, and displays it in the projection window of the target device. Exemplarily, the projection window of the target device switches the application projection interface corresponding to the second display data to the application projection interface corresponding to the third display data, completing the switching of the application interfaces of different applications (e.g., the first application and the second application) in the projection window.

[0266] 14 and 15 , the specific implementation of the aforementioned embodiments will be described in other embodiments, if the source device has activated the security authentication function.

[0267] In other embodiments, the source device can optionally enable a security authentication function to protect the privacy of the source device that has locked or turned off the screen. Before the projection window 201 displays the application interface 202, a security authentication interface can also be displayed, and a transition loading interface can also be displayed. In this case, the above step S201, that is, after the message center module receives the first operation of the user clicking the message notification control, the security authentication mechanism will be triggered first. For the specific implementation process, please refer to Figures 14, 15 and the following description.

[0268] As shown in FIG14 , the above method may further include:

[0269] S401: When the mobile phone is in a locked screen or off screen state, the message center module receives a second operation of the user clicking a message notification control.

[0270] It should be noted that the second operation here and the first operation in step S201 in Figure 11 are both operations in which the user clicks on the message notification control. In this embodiment, since a security authentication mechanism has been added, the steps shown in Figures 14 and 15 are added before triggering the establishment of the third communication connection in Figure 11.

[0271] It is understandable that when the phone is locked or off, it does not undergo user security authentication. In this case, if the PC is allowed to directly start screen casting without passing security authentication, the user's privacy will not be guaranteed. Therefore, in some embodiments, a security authentication mechanism will be triggered before starting screen casting from the PC to ensure user privacy.

[0272] Since the embodiment of the present application directly starts screen projection on the PC, the security authentication mechanism will be triggered on the PC, and the password authentication is distributed on the mobile phone. Therefore, the second security authentication module on the PC needs to send the authentication information (password) received on the PC to the first security authentication module on the mobile phone for password authentication. The mobile phone will return the authentication result to the PC. Therefore, in order to transmit the signaling required for security authentication between the mobile phone and the PC, a transmission channel needs to be established first.

[0273] S402: The message center module sends a message to the second communication module instructing to create a first session. The message carries the service channel name (Bluetooth channel) and the device identification code of the mobile phone.

[0274] Among them, illustratively, the service channel name can be a Bluetooth channel, the device identification code is the identification code of the mobile phone, and the above-mentioned instruction to create the first session is used to instruct the second communication module to establish a Bluetooth channel between the PC and the mobile phone corresponding to the device identification code.

[0275] S403: The second communication module establishes a second communication connection (Bluetooth connection) with the first communication module.

[0276] Exemplarily, the second communication connection may be a Bluetooth connection (BR connection), which provides a securely authenticated signaling transmission channel.

[0277] S404: The first communication module sends a message instructing the screen projection assistant to start.

[0278] S405, the screen projection assistant starts.

[0279] S406: The second communication module returns a message to the message center module indicating that the second communication connection is successfully established.

[0280] S407, the message center module sends a message instructing the second screen projection service module to start.

[0281] S408, the second screen projection service module is started.

[0282] S409, the second screen projection service module returns a message of successful startup to the message center module.

[0283] It should be noted that at this time, the screen projection assistant on the mobile phone and the second screen projection service module on the PC have been started, providing the necessary functions for the subsequent screen projection service.

[0284] In some embodiments, as shown in FIG15 , the method further includes:

[0285] S501: The message center module sends a security authentication request to the second security authentication module of the PC.

[0286] S502: The second security authentication module controls the display screen of the PC to display a security authentication window.

[0287] Exemplarily, a security authentication interface is displayed in the projection window 201, and the user can enter a password in the security authentication window.

[0288] It is understood that security authentication methods are not limited to password input for security authentication. For example, in the fingerprint unlocking scenario, fingerprints are collected for verification; in the voice unlocking scenario, voices are collected for verification. This is not limited here, and password unlocking is only used as an example.

[0289] S503: The second security authentication module receives the password input by the user through the security authentication window.

[0290] S504: The second security authentication module sends a password to the second communication module.

[0291] S505: The second communication module sends the password to the first communication module based on the second communication connection.

[0292] S506: The first communication module sends a password to the first security authentication module of the mobile phone.

[0293] It is understood that the security authentication method may also be fingerprint authentication by the user inputting a fingerprint, voice authentication by the user inputting a voice, etc., and is not limited thereto. The first security authentication module receives the password collected by the PC and compares it with the unlock password set by the user on the mobile phone. If the password collected by the PC matches the unlock password set on the mobile phone, the authentication is successful; otherwise, the authentication fails.

[0294] S507: The first security authentication module successfully performs password authentication.

[0295] If password authentication succeeds on the phone, a successful result is returned to the PC. If authentication fails on the phone, a failed result is returned to the PC. Optionally, upon receiving the failed result, the PC can prompt the user in the security authentication window that the password is incorrect. The user can re-enter the password and then initiate a security authentication request to the phone for password verification. Of course, you can set the authentication to suspend if the number of failed authentications exceeds a threshold for incorrect passwords. This threshold can be set by the user or left as the default value.

[0296] S508: The first security authentication module sends a message of successful authentication to the first communication module.

[0297] S509: The first communication module sends an authentication success message to the second communication module based on the second communication connection.

[0298] S510: The second communication module sends a message indicating successful authentication to the second security authentication module.

[0299] S511: The second security authentication module sends a message indicating successful authentication to the message center module.

[0300] S512: The message center module controls the display screen to display a transition loading window.

[0301] It is understandable that if the authentication is successful, the application screen projection steps of Figures 11 to 12 will be started. It takes a certain amount of time to successfully start the application screen projection, so there is a certain time span between displaying the security authentication window and displaying the application screen projection window. Optionally, in order to improve the user's perception and experience, after the security authentication is successful, the transition loading window can be started and displayed on the target device, which can prompt the user that the security authentication has passed and the application screen projection is being started. After the application screen projection steps of Figures 11 to 12 are successfully started, the screen projection window 201 displays the transition loading interface and switches to the application interface.

[0302] In other embodiments, after the target device receives a user instruction to project an application from the corresponding source device, it first determines whether security authentication is required. For example, this may determine whether the source device is projecting an application to the target device for the first time. Another example may determine whether the source device has an unlock password set. For details, please refer to the description in the previous embodiments.

[0303] The above describes in detail the screen projection method and examples of electronic devices provided in the embodiments of the present application. It is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0304] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each function can be divided into various functional modules, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0305] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0306] The electronic device provided in this embodiment is used to execute the above-mentioned screen projection method, and thus can achieve the same effect as the above-mentioned implementation method.

[0307] When integrated, the electronic device may also include a processing module, a storage module, and a communication module. The processing module may be used to control and manage the operation of the electronic device. The storage module may be used to support the execution of program code and data stored in the electronic device. The communication module may be used to support communication between the electronic device and other devices.

[0308] The processing module may be a processor or a controller. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module may be a memory. The communication module may specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, or a Wi-Fi chip.

[0309] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a device having the structure shown in FIG. 9 .

[0310] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the screen projection method of any of the above embodiments.

[0311] An embodiment of the present application also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement the screen projection method in the above-mentioned embodiment.

[0312] In addition, embodiments of the present application further provide a device, which may be a chip, component, or module. The device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions. When the device is running, the processor may execute the computer-executable instructions stored in the memory to cause the chip to perform the caller ID method in each of the above-mentioned method embodiments. The electronic device, computer-readable storage medium, computer program product, or chip provided in this embodiment is used to perform the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above and will not be repeated here.

[0313] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0314] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0315] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0316] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0317] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. The above embodiments are only used to illustrate the technical solution of the present application, not to limit it; although the present application is described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that it can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A screen projection method, characterized in that: Applied to a first device and a second device, the method includes: After the first device projects the screen onto the second device, the second device displays the first window; When the first device is in a screen-on state, the second device displays first image information from the first device in the first window, where the first image information corresponds to an application interface of a first application in the first device; After the first device switches to a locked screen or off screen state, the second device displays second image information from the first device in the first window, and the second image information also corresponds to an application interface of the first application in the first device.

2. The method according to claim 1, characterized in that After the first device switches to a locked screen or a screen-off state, the method further includes: The first device configures the first application not to sleep during its life cycle, and configures the application interface of the first application to be allowed to be drawn in a first display area during its life cycle, where the first display area is an invisible display area in the first device; The first device sends second image information of the first application to the second device, instructing the second device to display the second image information in the first window, where the second image information is data drawn in the first display area.

3. The method according to claim 1 or 2, characterized in that Before the first device projects the screen onto the second device, the method further includes: The first device receives a first message from a first application; The first device forwards the first message to the second device; The second device displays a first control, where the first control is used to display the first message; In response to a user operation on the first control, the second device sends a first request to the first device, where the first request includes a first identifier of the first application; The first device initiates screen projection to the second device in response to the first request.

4. The method according to claim 3, characterized in that After initiating screen projection to the second device, the method further includes: The first device creates a first display area and starts the first application; After the application interface of the first application is drawn in the first display area, the first device generates the first image information according to the image data in the first display area; The first device sends the first image information to the second device.

5. The method according to claim 4, characterized in that Before starting the first application, the method further includes: The first device marks the first application with a first tag; Before the first device configures the first application not to sleep during its life cycle and configures the application interface of the first application to be allowed to be drawn in the first display area during its life cycle, the method further includes: The first device determines that the first application has the first tag.

6. The method according to claim 2, characterized in that Before the first device switches to a locked screen or a screen-off state, the method further includes: In response to the user's instruction to lock the screen or turn off the screen, the first device only controls the display screen of the first device to power off or only displays the lock screen layer on the display screen, and the service running status corresponding to the display screen remains unchanged.

7. The method according to claim 2, characterized in that The first display area is marked with a second label. Before the first device switches to a locked screen or a screen-off state, the method further includes: When the first device detects that a condition for automatically turning off the screen is met, determining that the first display area has the second label; The first device only controls the display screen of the first device to power off or only displays the lock screen layer on the display screen, and the service running status corresponding to the display screen remains unchanged.

8. The method according to claim 2, characterized in that Before the first device switches to a locked screen or a screen-off state, the method further includes: The first device determines not to draw a lock screen layer in the first display area.

9. The method according to claim 2, characterized in that Before the first device switches to a locked screen or a screen-off state, the method further includes: The first device determines that the first display area does not mirror content in a second display area, the second display area being a display area visible in the first device.

10. A screen projection method, characterized in that: Applied to a first device, the method includes: The first device projects the screen onto the second device; When the first device is in a screen-on state, the first device sends first image information corresponding to the first application to the second device, instructing the second device to display the first image information on a first window; After the first device switches to the lock screen or screen-off state, the first device sends the second image information corresponding to the first application to the second device, instructing the second device to display the second image information on the first window, and the first image information and the second image information are both application interfaces of the first application.

11. The method according to claim 10, characterized in that After the first device switches to a locked screen or a screen-off state, the method further includes: The first device configures the first application not to sleep during its life cycle, and configures the application interface of the first application to be allowed to be drawn in a first display area during its life cycle, where the first display area is an invisible display area in the first device; The first device generates the second image information according to the content drawn in the first display area.

12. The method according to claim 10 or 11, characterized in that Before the first device projects the screen onto the second device, the method further includes: The first device receives a first message from a first application; The first device forwards the first message to the second device; After receiving the first request sent by the second device, screen projection to the second device is started, where the first request is request information for screen projection for the first application, and the first request carries a first identifier of the first application.

13. The method according to claim 12, characterized in that Before the first device sends the first image information corresponding to the first application to the second device, the method further includes: The first device creates a first display area and starts the first application; After the application interface of the first application is drawn in the first display area, the first device generates the first image information according to the image data in the first display area.

14. The method according to claim 13, characterized in that Before starting the first application, the method further includes: The first device marks the first application with a first tag; Before the first device configures the first application not to sleep during its life cycle and configures the application interface of the first application to be allowed to be drawn in the first display area during its life cycle, the method further includes: The first device determines that the first application has the first tag.

15. The method according to claim 11, characterized in that Before the first device switches to a locked screen or a screen-off state, the method further includes: In response to the user's instruction to lock the screen or turn off the screen, the first device only controls the display screen of the first device to power off or only displays the lock screen layer on the display screen, and the service running status corresponding to the display screen remains unchanged.

16. The method according to claim 11, characterized in that The first display area is marked with a second label. Before the first device switches to a locked screen or a screen-off state, the method further includes: When the first device detects that a condition for automatically turning off the screen is met, determining that the first display area has the second label; The first device only controls the display screen of the first device to power off or only displays the lock screen layer on the display screen, and the service running status corresponding to the display screen remains unchanged.

17. The method according to claim 11, characterized in that Before the first device switches to a locked screen or a screen-off state, the method further includes: The first device determines not to draw a lock screen layer in the first display area.

18. The method according to claim 11, characterized in that Before the first device switches to a locked screen or a screen-off state, the method further includes: The first device determines that the first display area does not mirror content in a second display area, the second display area being a display area visible in the first device.

19. An electronic device, characterized in that: The electronic device includes one or more processors and a memory; the memory is coupled to the processor, and the memory is used to store computer program code, the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the one or more processors are used to execute the method as described in any one of claims 10-18.

20. A computer storage medium, characterized in that The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 10 to 18.

Images