Screen projection system and electronic equipment

Abstract

The embodiment of the invention provides a screen projection system and electronic equipment, the system comprises first electronic equipment and second electronic equipment, an application message on the first electronic equipment flows to the second electronic equipment, and a corresponding message notification control is displayed on the second electronic equipment; the second electronic equipment receives an operation of a user on the message notification control at the first time, and sends a screen projection request to the first electronic equipment, so that an interface corresponding to the application message is displayed in a screen projection window of the second electronic equipment at the second time, and the interface comprises image information corresponding to the application message; the second time is later than the first time, and in a time interval between the first time and the second time, the first electronic equipment is in a screen locking state or a screen turn-off state. Therefore, when the first electronic equipment is in a screen locking or screen turn-off state, a user can directly operate on the second electronic equipment, so that the application on the first electronic equipment is pulled up for screen projection on the second electronic equipment.

Description

Screen projection system and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 19, 2022, with application number 202211282319.9 and invention name “A screen projection system and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the fields of communication technology and terminal technology, and in particular to a screen projection system and electronic device. Background Art

[0003] The collaborative work of multiple electronic devices brings convenience to users' lives and work. Taking collaborative notifications as an example, when a mobile phone receives a message, not only can the message notification be displayed on the mobile phone, but the message notification can also be transferred to the PC for display. This makes it convenient for users to view messages across devices and improves the efficiency of collaborative work between devices. Of course, it is not enough to simply transfer message notifications between devices. After users view the message notifications from their mobile phones on the PC, they often need to process the message notifications through the PC, which requires casting the phone screen to the PC.

[0004] However, when the phone is locked or off, an exception may occur when the user starts screen projection on the PC.

[0005] Summary of the Invention

[0006] The present application provides a screen projection system and electronic device. After a message flow from a first electronic device is transferred to a second electronic device, the user is allowed to directly start the application corresponding to the message on the second electronic device and project the screen on the second electronic device when the first electronic device is in a locked or off state.

[0007] In a first aspect, a screen projection system is provided, which includes a first electronic device and a second electronic device, the first electronic device including a first application; the first electronic device is used to receive first data of the first application; the first electronic device is used to send a first message including the first data to the second electronic device; the second electronic device displays a first control in response to receiving the first message; the second electronic device is used to receive a first operation of the user on the first control at a first time; the second electronic device sends a screen projection request to the first electronic device in response to the first operation; the first electronic device sends image information of the first application to the second electronic device in response to receiving the screen projection request, the image information of the first application including the first data; the second electronic device is used to display a window according to the received image information of the first application, the first interface of the first application is displayed in the window at a second time, the first interface including the image information of the first application; the second time is later than the first time, and during the time interval between the first time and the second time, the first electronic device is in a locked screen or screen off state.

[0008] When the first electronic device is in a locked or off-screen state, the user does not need to perform any operations on the first electronic device, but can directly operate on the second electronic device. That is, the first operation on the first control mentioned above can be clicking the first control, thereby launching the first application on the first electronic device and casting the screen on the second electronic device. The first application is an application installed on the first electronic device, and whether the second electronic device has the first application installed does not affect the implementation of the above system.

[0009] The first electronic device sends the first message to the second electronic device; in response to receiving the first message, the second electronic device is used to display a first control; in a first time interval, the first electronic device is in a locked screen or screen-off state; the second electronic device is also used to receive a first operation of the user on the first control in the first time interval; in response to the first operation, the second electronic device is also used to send a screen projection request to the first electronic device; in response to receiving the screen projection request, the first electronic device is also used to send data of the first application corresponding to the first message to the second electronic device; in response to receiving the data of the first application corresponding to the first message, the second electronic device is also used to display a screen projection window, which displays a first interface in the first time interval, and the first interface includes the first message.

[0010] In an embodiment of the present application, in the second time interval, the first electronic device is also used to display the main interface; the first electronic device is also used to receive a second operation from the user; in response to receiving the second operation, the first electronic device is in a locked screen or screen-off state in the first time interval, and the start time of the first time interval is the end time of the second time interval; the first electronic device is also used to receive a third operation from the user; in response to receiving the third operation, the first electronic device displays the main interface in a third time interval, and the start time of the third time interval is the end time of the first time interval; in the third time interval, the second electronic device is also used to display the first interface in the projection window.

[0011] In the above implementation, the first electronic device is first locked or turned off while displaying the main interface. Subsequently, while the first electronic device is locked or turned off, a first message from the first application is received and transmitted to the second electronic device. After the above screen casting operation, the first interface is projected to the second electronic device. Subsequently, the first electronic device is turned on or unlocked, i.e., the third operation described above, at which point the first electronic device displays the main interface, while the second electronic device's projection window displays the first interface.

[0012] In combination with the first aspect, in some implementations of the first aspect, the first electronic device also includes a second application, and the first application is different from the second application; the first electronic device is also used to receive an operation by a user to open the second application; in response to the user's operation to open the second application, the first electronic device is also used to display the second interface of the second application in a fourth time interval, and the start time of the fourth time interval is the end time of the third time interval; in the fourth time interval, the second electronic device is also used to display the first interface in the projection window.

[0013] In the above implementation, when the projection window of the second electronic device displays the first interface, the first electronic device can open other applications different from the first application, such as the second application. At this time, the projection window of the second electronic device still displays the first interface and will not change with the change of the display interface on the first electronic device, thereby realizing heterogeneous screen projection.

[0014] In other possible embodiments, in the fourth time interval, the second electronic device is further used to display a third interface in the projection window, where the third interface is the interface of the first application, and the third interface is different from the first interface.

[0015] Among them, the third interface and the first interface are both interfaces of the first application. That is to say, after the first application is projected to the second electronic device, the first application can be operated in the projection window of the second electronic device to switch between different interfaces.

[0016] In an embodiment of the present application, the first electronic device also includes a third application, which is different from the first application; the first electronic device is also used to receive a second message of the third application; the first electronic device is also used to send the second message to the second electronic device; in response to receiving the second message, the second electronic device is also used to display a second control; the second electronic device is also used to receive a fourth operation of the user on the second control in the first time interval; in response to the fourth operation, the second electronic device is also used to send a screen projection request to the first electronic device; in response to receiving the screen projection request, the first electronic device sends the data of the third application corresponding to the second message to the second electronic device; in response to receiving the data of the third application corresponding to the second message, in the first time interval, the second electronic device is also used to display the fourth interface in the screen projection window, and the fourth interface includes the second message.

[0017] In the above implementation, after the first application is projected to the second electronic device, if the first electronic device receives a message from another application, such as a second message from a third application, the user can also switch the projection through the above operation and project the second application to the second electronic device.

[0018] In combination with the first aspect, in some implementations of the first aspect, the second electronic device is further used to display the first control on the desktop; or, the second electronic device is further used to display the first control in the message list window, and the first control is a message card corresponding to the first message.

[0019] The first control is a message card corresponding to the first message. Besides being displayed separately on the desktop, the message card also has another possibility: if the first control remains displayed on the desktop for a certain period of time without receiving any user action, it disappears and a message card corresponding to the first message is generated in the message list. If the user wishes to perform an action on the first message after the first control disappears, they can open the message list window and find the message card corresponding to the first message in the message list window, thus achieving the same effect as if the user had operated the first control.

[0020] In some possible implementations, the first operation is an operation of the user clicking the first control; or, the first operation is an operation of the user dragging the first control.

[0021] In other possible implementations, the second electronic device is further used to receive the first operation and display a security authentication interface in the screen projection window; upon receiving an operation in which the user enters authentication information in the security authentication interface, the second electronic device is further used to send the authentication information to the first electronic device; upon receiving the authentication information, the first electronic device is further used to determine that the authentication information meets a preset value and send a message of successful authentication to the second electronic device; in response to receiving the message of successful authentication, the second electronic device sends a screen projection request to the first electronic device.

[0022] In the above implementation, when the first electronic device is locked or off, it does not undergo user security authentication. In this case, if the second electronic device is allowed to directly start screen projection without security authentication, the user's privacy security will not be guaranteed. Therefore, in some embodiments, a security authentication mechanism is triggered before the second electronic device starts screen projection to ensure user privacy security.

[0023] In combination with the first aspect, in certain implementations of the first aspect, the second electronic device is further used to receive the user's operation on the first message on the first interface, and the second electronic device sends an instruction to the first electronic device instructing the first electronic device to perform the operation.

[0024] Among them, after the first application is projected to the second electronic device, the user can directly reply to the first application, make audio and video calls, and perform other operations on the second electronic device. The second electronic device will send all these operation instructions to the first electronic device, which will be executed by the first electronic device, thereby realizing the control of the projected application on the second electronic device.

[0025] In a second aspect, a screen projection method is provided, which is applied to a first electronic device, wherein the first electronic device includes a first application; the first electronic device receives a first message of the first application; the first electronic device sends the first message to a second electronic device through a first communication connection; in a first time interval, the first electronic device is in a locked screen or screen-off state; in response to receiving a screen projection request sent by the second electronic device, the first electronic device sends the data of the first application corresponding to the first message to the second electronic device through a second communication connection, so that in the first time interval, the second electronic device displays the first interface in the screen projection window; wherein the first interface includes the first message.

[0026] In combination with the second aspect, in certain implementations of the second aspect, in the first time interval, after the first electronic device receives the screen projection request, the method includes the first electronic device maintaining one or more activities of the first application in the OnResume state; the first electronic device controls one or more activities of the first application not to enter the sleep process.

[0027] Among them, the first electronic device keeps one or more activities of the first application in the OnResume state, and at the same time controls one or more activities of the first application not to enter the sleep process, so that the one or more applications can be visible to the user and the user can operate after the screen is projected to the second electronic device.

[0028] In some possible implementations, the first electronic device includes an asynchronous processor, an activity recorder, and a root window container. In the first time interval, after the first electronic device receives the screen projection request, the method also includes the asynchronous processor marking one or more activities of the first application with a first label; the activity recorder determines that one or more activities of the first application have the first label, and keeps the one or more activities of the first application in the OnResume state; the root window container determines that one or more activities of the first application have the first label, and controls the one or more activities of the first application not to enter the sleep process.

[0029] Among them, the first electronic device can filter out the activities corresponding to the first application from the activities corresponding to other applications by marking the activities corresponding to the first application with a specific label, so as to perform the above-mentioned screen projection operation only on the activities of the first application that need to be projected.

[0030] In combination with the second aspect, in certain implementations of the second aspect, in the first time interval, the first electronic device includes a virtual display area and a visible display area. Before the first electronic device sends the first interface to the second electronic device, the first electronic device marks the virtual display area with a second label; the first electronic device draws a first image in the virtual display area, and the first image is the interface of the first application corresponding to the first message; the first electronic device draws a second image in the visible display area, and the second image is the interface corresponding to the first electronic device when the first electronic device is in the locked screen or screen-off state; the first electronic device determines that the virtual display area has the second label, obtains the first image to obtain the first interface, and the first interface does not include the second image.

[0031] Among them, the first electronic device marks the virtual display area with a specific label, thereby indicating that the image drawn in the virtual display area will not be synthesized with the images in other display areas, so that the interface projected to the second electronic device is only related to the image in the virtual display area, and will not change with the changes of the images in other display areas, nor will the lock screen layer be covered on the projection interface.

[0032] In some possible implementations, the first electronic device includes a display screen, and the method also includes the first electronic device displaying a main interface in a second time interval; the first electronic device receives a second operation from the user; in response to receiving the second operation, the first electronic device is in a locked screen or screen-off state at a first time, and the first time is the end time of the second time interval and the start time of the first time interval; the first electronic device receives a third operation from the user; in response to receiving the third operation, the first electronic device draws the main interface in the visible display area and displays the main interface in a third time interval, and the start time of the third time interval is the end time of the first time interval; in the third time interval, the first electronic device determines that the virtual display area has the second label, and does not obtain the main interface of the visible display area and send it to the second electronic device, so that the second electronic device displays the first interface in the projection window.

[0033] In combination with the second aspect, in certain implementations of the second aspect, before the first electronic device receives the request to start screen projection, the method also includes the first electronic device receiving authentication information sent by the second electronic device through a third communication connection; the first electronic device determines that the authentication information meets the preset value, and sends a message of successful authentication to the second electronic device through the third communication connection, so that the second electronic device sends a screen projection request to the first electronic device.

[0034] In some possible implementations, the method includes the first electronic device receiving an instruction from the second electronic device and executing an operation of the user on the first message on the first interface.

[0035] According to a third aspect, a method is provided, which is applied to a first electronic device and a second electronic device, wherein before the second electronic device establishes the first communication connection with the first electronic device, the first electronic device establishes a second communication connection with the second electronic device; the first electronic device receives the first message; the first device sends the second message to the second electronic device through the second communication connection based on the first message; in response to receiving the second message, the second electronic device displays the first control based on the second message; when the first electronic device is in a locked screen or screen-off state, the second electronic device receives the first operation.

[0036] In an embodiment of the present application, after the second electronic device receives the first operation, the second electronic device determines that the first application is projected on the second electronic device for the first time, and the second electronic device establishes a third communication connection with the first electronic device, and the third communication connection is used for security authentication; the projection window displays the security authentication interface of the first electronic device.

[0037] In combination with the third aspect, in certain embodiments of the third aspect, after the second electronic device receives the first operation, the second electronic device determines that this is not the first time that the first application is projected on the second electronic device, and the current time is longer than the preset time from the last successful authentication, and the second electronic device establishes a third communication connection with the first electronic device; the projection window displays the security authentication interface of the first electronic device.

[0038] In some possible implementations, after the second electronic device receives the user's operation of inputting authentication information in the security authentication interface, the second electronic device sends the authentication information to the first electronic device through the third communication connection; the first electronic device determines that the authentication information meets the preset value, and sends a message of successful authentication to the second electronic device through the third communication connection.

[0039] In other possible embodiments, after the second electronic device receives the user's first operation on the first control, the second electronic device determines that the first application is set with an application lock, and the second electronic device does not display the projection window of the first electronic device.

[0040] In combination with the third aspect, in certain embodiments of the third aspect, the first electronic device includes a screen projection assistant module, a first virtualization service module, and a first screen projection service module. After the first electronic device receives the screen projection request, the screen projection assistant module binds the screen projection service; the screen projection assistant module sends a first indication message for initialization to the first virtualization service module; the first virtualization service module completes initialization according to the first indication message; the first virtualization service module sends a second indication message to start the screen projection service to the first screen projection service module; the first screen projection service module starts the screen projection service according to the second indication message; the first screen projection service module sends a message to the screen projection assistant module that the screen projection service has been started.

[0041] In some possible implementations, the first electronic device creates a virtual display area and marks the first label for the virtual display area. After the first screen projection service module starts the screen projection service, it creates the virtual display area and marks the first label for the virtual display area.

[0042] In other possible implementations, the first electronic device also includes a multi-screen management module and a system interface module, and the second electronic device includes a second screen projection service module and a message center module. After the first screen projection service module sends a message to the screen projection assistant module that the screen projection service has been started, the screen projection assistant module sends a message to the message center module that the screen projection service has been started through the first communication connection; the message center module sends a request to start application screen projection to the system interface module through the first communication connection; the system interface module sends a third indication message to start the first application screen projection to the multi-screen management module based on the request to start application screen projection; upon receiving the third indication message, the multi-screen management module puts all activities of the first application in the foreground visible state, and skips the dormant state for all activities of the first application; the multi-screen management module obtains the first display data corresponding to the first application; and the screen projection management module sends the first display data to the second screen projection service module through the first communication connection.

[0043] In combination with the third aspect, in certain embodiments of the third aspect, the multi-screen management module places all activities of the first application in a foreground visible state, and skips the dormant state for all activities of the first application, and the multi-screen management module marks all activities of the first application with a second label; the multi-screen management module places the activities marked with the second label in a foreground visible state, and skips the dormant state for the activities marked with the second label.

[0044] In combination with the third aspect, in certain embodiments of the third aspect, before the multi-screen management module obtains the first display data corresponding to the first application, the method includes: the multi-screen management module sends a fourth indication message to the first application; the first application is started according to the fourth indication message; the first application sends a message to the multi-screen management module that the first application has been started; after receiving the message that the first application has been started, the multi-screen management module draws the first display data in the virtual display area.

[0045] In an embodiment of the present application, after the multi-screen management module sends the first display data to the second screen projection service module through the first communication connection, the method includes: the second screen projection service module receives the first display data, and renders the first interface based on the first display data; the second screen projection service module sends the first interface to the message center module; after receiving the first interface, the message center controls the display screen of the second electronic device to display the first interface in the projection window.

[0046] In some possible embodiments, the first electronic device also includes a first communication module, the second electronic device also includes a second communication module, and the second electronic device establishes a first communication connection with the first electronic device, specifically including: the message center module sends a fifth indication information to the second communication module; the second communication module establishes the first communication connection with the first communication module according to the fifth indication information; the second communication module sends a message to the message center indicating that the first communication connection is successfully established.

[0047] In some other possible embodiments, before the screen projection assistant module is bound to the screen projection service, the method includes: the first communication module sends sixth indication information to the screen projection assistant module; and the screen projection assistant module is started according to the sixth indication information.

[0048] In combination with the third aspect, in certain embodiments of the third aspect, before the second electronic device sends a screen projection request to the first electronic device through the first communication connection, it specifically includes: the message center obtains the session key and the IP address of the first electronic device through the first communication connection; the message center sends a seventh indication message to the second screen projection service module, and the seventh indication message carries the session key and the IP address of the first electronic device; the second screen projection service module completes initialization according to the seventh indication message.

[0049] In an embodiment of the present application, before the first electronic device receives the screen projection request, it specifically includes: the second screen projection service module sends the screen projection request to the second communication module through the first communication connection; the second communication module sends the screen projection request to the first communication module through the first communication connection; the first communication module sends the screen projection request to the screen projection assistant module.

[0050] In combination with the third aspect, in certain embodiments of the third aspect, after the first electronic device receives the first message, it specifically includes: the first application sends the second message to the system interface module based on the first message; the system interface module sends the second message to the message center module through the second communication connection; after receiving the second message, the message center module controls the display screen of the second electronic device to display the first control based on the second message; after receiving the first operation of the user on the first control, the message center module sends a fifth indication message to the second communication module.

[0051] In some possible implementations, after the message center module receives the first operation, the method further includes: the message center module controls the display screen of the second electronic device to display the transition loading interface in the projection window.

[0052] In other possible implementations, the second electronic device also includes a second security authentication module. After the message center module receives the first operation, the method also includes: the message center sends an eighth indication information to the second security authentication module; the second security authentication module determines, based on the eighth indication information, that the first application is on the second electronic device for the first time, and the second security authentication module sends a ninth indication information to the second communication module; the second communication module establishes the third communication connection with the first communication module based on the ninth indication information.

[0053] In combination with the third aspect, in certain embodiments of the third aspect, after the message center module receives the first operation, the method also includes: the message center sends an eighth indication message to the second security authentication module; the second security authentication module determines, based on the eighth indication message, that this is not the first time that the first application is projected on the second electronic device, and the current time is longer than the preset value from the last successful authentication, and the second security authentication module sends the ninth indication message to the second communication module; the second communication module establishes the third communication connection with the first communication module based on the ninth indication message.

[0054] In combination with the third aspect, in certain embodiments of the third aspect, after the third communication connection is established, it specifically includes: the second communication module sends a message that the third communication connection is successfully established to the message center module; the message center sends a tenth indication message to the second security authentication module; and the second security authentication module controls the display screen of the second electronic device to display the security authentication interface in the projection window according to the tenth indication message.

[0055] In an embodiment of the present application, the first electronic device also includes a first security authentication module. After the second security authentication module receives the operation of the user entering authentication information on the security authentication interface, it specifically includes: the second security authentication module sends the authentication information to the first security authentication module through the third communication connection; the first security authentication module determines that the authentication information meets the preset value, and sends a successful authentication message to the second security authentication module through the third communication connection; the second security authentication module sends the successful authentication message to the message center module; after receiving the successful authentication message, the message center module controls the display screen of the second electronic device to display the transition loading interface in the projection window.

[0056] In combination with the third aspect, in certain embodiments of the third aspect, the multi-screen management module includes an asynchronous processor, an activity launcher, an activity stack manager, a lock screen controller, an activity recorder, a task stack, and a root window container. The multi-screen management module puts all activities of the first application in a foreground visible state and skips all activities of the first application in a dormant state, specifically including: the asynchronous processor marks some or all activities of the first application with the second label; the asynchronous processor sends a request to the activity launcher to start an activity with the second label; in response to receiving the first command, the activity launcher sends a request to start application screen projection to the activity stack manager; in response to receiving the request to start application screen projection, the activity stack manager calls the lock screen controller to perform a first action, the first action including putting the activity marked with the second label in a foreground visible state; the lock screen controller puts the activity marked with the second label in a foreground visible state. The lock screen controller sends a third command to the activity recorder; after receiving the third command, the activity recorder controls the activity marked with the second label to remain visible in the foreground; the activity recorder sends a fourth command to the task stack, and the fourth command is used to instruct the task stack to check the status of the activity at the top of the task stack; after receiving the fourth command, the task stack determines that the activity at the top of the task stack is in the foreground visible state, and sends a fifth command to the root window container; after receiving the fifth command, the root window container skips the sleep state of the activity marked with the second label.

[0057] In combination with the third aspect, in certain embodiments of the third aspect, the first message corresponds to a first activity of the first application, the first activity is marked with the second tag by the asynchronous processor, and the first activity is located at the top of the task stack.

[0058] In some possible implementations, the multi-screen management module also includes an activity stack management server and a layer synthesizer. Before the multi-screen management module obtains the first display data corresponding to the first application, it specifically includes: after receiving the first command, the activity launcher sends a fourth indication information to the first application; the first application is started according to the fourth indication information; the first application sends a message to the activity stack management server that the first application has been started; the activity stack management server sends a sixth command to the layer synthesizer; the layer synthesizer draws the application layer corresponding to the first application in the virtual display area; the layer synthesizer obtains the first display data according to the application layer.

[0059] In a fourth aspect, an electronic device is provided, which includes one or more processors; one or more memories; the one or more memories store one or more computer programs, and the one or more computer programs include instructions, which, when executed by the one or more processors, enable the electronic device to execute any method that may be implemented in the above-mentioned second aspect.

[0060] In a fifth aspect, a computer-readable storage medium is provided, comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method described in the second aspect.

[0061] In a sixth aspect, a computer program product is provided, which includes a computer program code. When the computer program code runs on an electronic device, the electronic device executes the method described in the second aspect.

[0062] In a seventh aspect, a chip is provided for executing instructions. When the chip is running, the chip executes the method described in the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] FIG1 is a schematic diagram of a message flow scenario provided in an embodiment of the present application.

[0064] Figure 2 is a schematic diagram of the interface display of a heterogeneous screen projection provided in an embodiment of the present application.

[0065] Figure 3 is a schematic diagram of the interface display of a heterogeneous screen projection provided in an embodiment of the present application.

[0066] Figure 4 is a schematic diagram of the principle of heterogeneous screen projection provided in an embodiment of the present application.

[0067] Figure 5 is a schematic diagram of a screen projection scenario in the screen-off state provided in an embodiment of the present application.

[0068] Figure 6 is a schematic diagram of a screen projection scenario in a locked screen state provided in an embodiment of the present application.

[0069] FIG7 is a schematic diagram of another scenario of screen projection in the screen-off state provided in an embodiment of the present application.

[0070] FIG8 is a schematic diagram of a screen projection operation provided in an embodiment of the present application.

[0071] FIG9 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0072] FIG10 is a software architecture diagram of an electronic device provided in an embodiment of the present application.

[0073] FIG11 is a diagram showing a specific implementation process of a screen projection method provided in an embodiment of the present application.

[0074] Figure 12 is a specific implementation process diagram of another screen projection method provided in an embodiment of the present application.

[0075] Figure 13 is a specific implementation process diagram of another screen projection method provided in an embodiment of the present application.

[0076] Figure 14 is a specific implementation process diagram of another screen projection method provided in an embodiment of the present application.

[0077] Figure 15 is a specific implementation process diagram of another screen projection method provided in an embodiment of the present application.

[0078] Figure 16 is a specific implementation process diagram of another screen projection method provided in an embodiment of the present application.

[0079] FIG17A is a schematic diagram of a principle provided in an embodiment of the present application.

[0080] FIG17B is another schematic diagram of a principle provided in an embodiment of the present application.

[0081] Figure 18 is a flowchart of a screen projection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0082] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0083] It should be understood that the terms "first," "second," and the like in the specification, claims, and drawings of this application are used to distinguish between different objects, rather than to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0084] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0085] Illustratively, an embodiment of the present application provides a system including multiple electronic devices, such as device 1, device 2, and device 3. The electronic devices in the above system (ie, device 1, device 2, device 3) may belong to the same local area network.

[0086] The local area network may be a wireless communication network, such as a wireless local area network (WLAN), wireless fidelity point to point (Wi-Fi P2P), a Bluetooth network, a ZigBee network, an infrared (IR) or near field communication (NFC) network. Furthermore, the local area network may be a wired communication network, such as a communication network established by a video graphics array (VGA), a digital visual interface (DVI), a high definition multimedia interface (HDMI), or a data transmission line. Furthermore, the local area network may be a remote network.

[0087] After devices 1, 2, and 3 have mutually recognized trust, the system consisting of devices 1, 2, and 3 can be called a trust ring. The process of mutual trust recognition between devices can refer to relevant technologies. For example, the process can be determined based on one or more factors, such as whether the devices have logged into the same system account, whether they have been authorized, and whether near-field communication is used. The details are not detailed here.

[0088] It can be understood that in the same trust ring, multiple electronic devices can safely share data. The shared data includes but is not limited to message reminders, message records, user information of other electronic devices, device types, running status of each application, identification of each application, synchronization of identification of recently used applications, etc.

[0089] In this way, a variety of collaborative services can be realized within the trust ring, such as collaborative calls, collaborative notifications, screen projection and other services.

[0090] Taking collaborative notifications as an example, in a trusted ring, when device 1 receives a message notification, it can not only display it on device 1 but also forward it to device 2 and / or device 3 for display. This makes it easier for users to view messages across devices and improves the efficiency of inter-device collaboration. Similarly, device 2 can forward received message notifications to device 1 and / or device 3, and device 3 can also forward received message notifications to device 1 and / or device 2.

[0091] Of course, it is not enough to just transfer messages between devices. If a user views a message notification from device 1 on device 2, he or she often needs to process the message notification through device 2, which requires casting the screen of device 1 to device 2.

[0092] Generally speaking, a screen projection scenario involves a source device and a target device. The source and target devices communicate via a protocol, enabling the content on the source device to be displayed on the target device. This is known as screen projection. Common protocols include the DLNA (Digital Living Network Alliance) protocol, Chromecast protocol, Miracast protocol, and Airplay protocol. Source devices can be phones, tablets, PCs, watches, and more. Target devices can be tablets, PCs, TVs, and more.

[0093] It is understandable that the source device and the target device may also have other names, for example, the source device may be called an active screen projection device, and the target device may be called a passive screen projection device; or, the source device may be called a first electronic device, and the target device may be called a second electronic device, etc. This application does not limit this.

[0094] As shown in Figure 1, the source device is a mobile phone and the target device is a PC. After the mobile phone and PC are on the same local area network and have established mutual trust, the mobile phone and PC belong to the same trust circle. In other words, a communication channel for message flow can be established between the mobile phone and PC.

[0095] In some embodiments, a mobile phone may have multiple types of applications installed in it. During operation of the mobile phone, the mobile phone may receive application messages pushed by various applications. The application messages come from a message server corresponding to the application.

[0096] For example, a social application may be installed in the mobile phone, and the mobile phone may receive application messages pushed by the social application, such as a message "Are you going for a run today?" sent by Xiao Ming, a contact in the social application.

[0097] After receiving an application message pushed by an application, if the phone is unlocked and the screen is on, the phone can display a prompt message on the current interface. For example, when receiving an application message pushed by a social application, if the current interface of the phone is the main interface 101, a notification bubble 102 is displayed on the main interface 101, and the notification bubble 102 is used to display the application message received by the phone.

[0098] Then, the mobile phone can transfer the above application message (for example, "Are you going for a run today?" pushed by the social application) to the PC through the communication channel used for transferring messages between the mobile phone and the PC.

[0099] In some embodiments, after the PC receives an application message from a mobile phone, it can display a prompt message for the above application message. Among them, the above prompt message can be displayed on the desktop of the PC in the form of a message notification control. For example, the above message notification control can be a floating window, a message bubble, a message card, etc. Of course, in the embodiment of the present application, there is no specific limitation on its display form. For example, as shown in Figure 1, a message card 104 is displayed on the desktop 103, and the message card 104 is used to display the relevant content of the above application message. Among them, the relevant content can be that the application message comes from a mobile phone, or that the application message comes from a social application, or the main content of the application message, etc. It can be understood that in the embodiment of the present application, there is no specific limitation on the display position of the message card 104.

[0100] In the scenario shown in FIG1 , if the user is using a PC, the message card 104 displayed on the desktop 103 can remind the user that the mobile phone has received an application message. In this way, even if the user is not currently using the mobile phone or the mobile phone is not with him, he can still understand the brief content of the application message and avoid missing the message.

[0101] In some embodiments, the message card 104 may have a display time limit. For example, the display time limit is preset to duration 1. In response to the application message forwarded by the mobile phone, the PC begins to display the message card 104. If the display time of the message card 104 exceeds duration 1 and the user does not select the message card 104, the PC may cancel the display of the message card 104.

[0102] For example, when the PC's display screen is a touch screen, the PC detects that the user has made contact with display area 1 of the touch screen, and the PC determines that the user has indicated the selection of message card 104. Display area 1 is the area on the touch screen that displays message card 104. For another example, when a mouse is connected to the PC, the PC detects that the mouse cursor is within display area 1 and a confirmation command is input, and the PC determines that the user has indicated the selection of message card 104.

[0103] In some embodiments, after the PC displays the message card 104, the user can process the application message by operating the PC, or can choose to process the application message by directly operating the mobile phone.

[0104] The method of directly operating the mobile phone to process application messages can refer to related technologies. For example, in response to the user clicking on the notification bubble 102, the mobile phone can run a social application in the foreground and display the application interface containing the above social message. In this way, the user can reply to the application message by operating on the above application interface. In another example, the mobile phone responds to the user's instruction to launch the social application and displays the application interface of the social application. In this way, the user can view the above application message or reply to the application message by operating on the application interface provided by the social application.

[0105] In addition, choosing to operate the PC to process application messages will trigger screen projection between the mobile phone and the PC. For example, the user can select the message card 104 on the PC. After the PC can detect that the user has selected the message card 104, the PC can remotely control the mobile phone to run the social application. After the social application enters the running state, the PC can display the projection window on the desktop 103. After the PC displays the projection window, the user can control the mobile phone through the projection window in the PC, such as operating the mobile phone to process the above-mentioned application messages.

[0106] The above-mentioned screen projection window can be used to synchronously display the current interface of the mobile phone. For example, when the current interface of the mobile phone is the application interface of a social application, the screen projection window also displays the application interface, which is a same-source screen projection. The current interface of the mobile phone can also be displayed asynchronously. For example, when the mobile phone displays the main interface, the screen projection window displays the application interface, which is a different-source screen projection.

[0107] Please refer to Figure 2, which introduces the interface display of heterogeneous screen projection in the above scenario.

[0108] As an implementation method, after the PC determines that the user has selected the message card 104, heterogeneous screen projection can be achieved through data exchange between the PC and the mobile phone. As shown in Figure 2, after the PC determines that the user has selected the message card 104, heterogeneous screen projection is initiated. The main interface 101 is still displayed on the mobile phone's display, and the projection window 201 on the PC's display displays the application interface 202 corresponding to the above application message.

[0109] After that, the content displayed in the projection window will not change synchronously with the content displayed on the mobile phone display. For example, referring to Figure 3, the mobile phone display switches to displaying the information application interface 303, while the projection window 201 on the PC display still displays the application interface 201 corresponding to the above application message.

[0110] Please refer to Figure 4, which is a schematic diagram of the principle of heterogeneous screen projection in Figures 2 and 3 above. In the heterogeneous screen projection scenario, when the source device projects the screen to the target device, the display information on the target device is not synchronized with the display information on the foreground of the source device. The window of application B is displayed on the source device. When the source device projects the window of application B to the target device, the window of application B is displayed in the created virtual display area (VirtualDisplay). Among them, the content of the virtual display area is invisible to the user and can be used for projection to the target device for display. In addition to the virtual display area, the source device also includes a user-visible display area. When the source device projects the window of application B to the target device, the user-visible display area displays the window of application A, that is, the source device displays the window of application A in the foreground. Assume that the user-visible area on the source device switches from the window of application A to the window of application C, that is, the source device displays the window of application C in the foreground. At this time, the window of application B is still displayed on the target device.

[0111] It can be understood that compared with the same-source screen projection, the heterogeneous screen projection method has better isolation. For example, the heterogeneous screen projection method can provide users with independent control screens (i.e., the display screen of the source device and the display screen of the target device) to handle different applications. In this way, after the user projects the window of application B to the target device for display, the source device operates application A, or switches the window of application A to the window of application C, and the user can still operate application B on the target device. Therefore, heterogeneous screen projection releases the control of the display information of the source device over the display information of the target device, thereby enhancing the reusability of multiple devices and allowing users to operate different applications on different devices through projection. In the same-source screen projection method, the content displayed in the projection window changes synchronously with the content displayed on the mobile phone display. In the embodiment of the present application, the heterogeneous screen projection scenario is described as an example.

[0112] Taking Figures 1 and 2 above as an example, it describes that when the mobile phone is in the screen-on and unlocked state, after the mobile phone receives a message notification, it transfers it to the PC. Then, the user operates the PC to select the message card that displays the message notification, triggering the process of heterogeneous screen projection from the mobile phone to the PC.

[0113] In the heterogeneous screen projection scenario, as shown in Figure 5, in the related art, when the mobile phone is in the screen-off state, the mobile phone display is in a black screen state. At this time, if the mobile phone receives an application message from a social application, the mobile phone can briefly light up the screen and display a notification bubble, or it can not display the notification bubble and continue to remain in a black screen state. It can be understood that the above-mentioned black screen state refers to the state where the mobile phone display is not lit or only displays the screen saver image.

[0114] As shown in Figure 6, when the phone is in the lock screen state, the phone display can display the lock screen interface or also be in a black screen state. At this time, if the phone receives an application message from a social application, the phone can display a notification bubble on the lock screen interface.

[0115] In the scenarios shown in Figures 5 and 6, the mobile phone can also transfer the application message to the PC. In this way, the PC can still display the message card 104 on the desktop 103.

[0116] Of course, in the scenarios shown in Figures 5 and 6, while the PC is displaying message card 104, even if the PC determines that the user has indicated that message card 104 has been selected, the phone's screen remains black or displays the lock screen. It is understandable that while the screen remains black or the lock screen is displayed, the phone cannot run the social application in the foreground, that is, it cannot display the application interface of the social application.

[0117] It should be noted that the screen-off described in this article specifically refers to the situation where the screen is locked and off. At this time, the source device displays a black screen interface and the display screen is not lit, as shown in Figure 5; the screen-locked described in this article specifically refers to the situation where the screen is locked but the screen is on. At this time, the source device displays the lock screen interface, and the lock screen interface may include a lock logo, as shown in Figure 6.

[0118] Correspondingly, as shown in FIG5 , even if the PC displays the projection window corresponding to the mobile phone, the projection window also displays a black screen interface.

[0119] Correspondingly, as shown in FIG6 , even if the PC displays the projection window corresponding to the mobile phone, the projection window also displays the lock screen interface.

[0120] As can be seen in the scenarios shown in Figures 5 and 6, regardless of whether the projection window displays a black screen or a lock screen, the user cannot remotely control the phone through the PC and process the aforementioned application messages. In other words, the user can only choose to unlock the phone and directly operate the phone to complete the processing of the application message.

[0121] Corresponding to Figure 5, if the user wants to display the screen projection window of the application interface on the PC, the user can first turn on the screen of the mobile phone and unlock it, and then select the message card 104 on the PC to start the heterogeneous screen projection. It is understandable that the screen projection scene after the mobile phone is unlocked can be referred to Figure 2, which will not be described in detail here.

[0122] Corresponding to Figure 6, if the user wants to display the screen projection window of the application interface on the PC, the user can first unlock the phone and keep the screen on, and then select the message card 104 on the PC to start the heterogeneous screen projection. It can be understood that the screen projection scene after the phone is unlocked can be referred to Figure 4, which will not be described in detail here.

[0123] That is, when the mobile phone is in the screen-off or locked state, there is such a scenario: when the user is using a PC to play games or work in the study or living room at home, the mobile phone is placed in another room; when a contact (Xiao Ming) on ​​a social application sends a message to the user's mobile phone, the user can immediately know on the PC that his mobile phone has received a message notification. However, in this scenario, the user is often focused on playing games or working on the PC, and does not want to get up and go to the room to get the mobile phone to check or reply to messages, but wants to start the application window displaying the message on the PC (that is, the screen projection window corresponding to the mobile phone) and process the message through the PC. However, since the mobile phone in the other room is in the lock screen state at this time, based on the existing technology, the user cannot directly start the application window displaying the message on the PC. The screen projection window must first be unlocked in the room and put in the bright screen state before the corresponding screen projection window of the mobile phone can be started on the PC.

[0124] It can be seen that when the mobile phone is in the locked screen or off screen state, it is very inconvenient and time-consuming for the user to start the screen projection on the PC. Of course, in the related art, when the mobile phone is in the off screen and locked screen state at the same time, similar problems also exist, which will not be repeated here.

[0125] As can be seen, under normal circumstances, when the source device is in the locked or off state, the user cannot directly launch the application screen projection window displaying the message on the target device. Instead, the source device must be unlocked and the screen is in the bright state before the application screen projection window displaying the message can be launched on the target device. Otherwise, a black screen window or a lock screen interface window will be displayed on the target device. This increases the user's operation process and further reduces the screen projection efficiency when the source device is not with the user, giving the user a cumbersome and inefficient screen projection experience.

[0126] It should be noted that the application screen projection window refers to the screen projection window displayed on the target device when the application on the source device is projected.

[0127] In order to solve the above problems, when the source device is locked or in the off state, the user can start the application screen projection window on the target device. The embodiments of the present application provide a screen projection method, electronic device and system to provide users with a convenient and efficient screen projection experience.

[0128] Specifically, in the scenario where the above-mentioned source device is in a locked or off-screen state, according to a screen projection method provided in this embodiment, the user can directly operate on the target device to realize the projection of the application window corresponding to the flow message on the target device. Based on the projection of the application window on the target device, the function of multi-screen collaborative interaction across devices is realized through the coordination of control and input. In this way, the user can focus on the operation of the target device without having to operate the source device in the locked or off-screen state, which reduces the user operation process of starting the projection, improves the efficiency of human-computer interaction in starting the projection, and enhances the user experience.

[0129] For example, please refer to Figure 7, which is a schematic diagram of a screen projection method provided by an embodiment of the present application. The embodiment of the present application can realize processes such as mobile phone screen projection to PC, mobile phone screen projection to tablet computer, and tablet computer screen projection to PC. The following is an example of the source device being a mobile phone and the target device being a PC.

[0130] After the mobile phone and PC belong to the same local area network and complete mutual trust recognition, the mobile phone and PC belong to the same trust ring, that is, a communication channel for message flow can be established between the mobile phone and PC.

[0131] For example, a social application may be installed in the mobile phone, and the mobile phone may receive application messages pushed by the social application, such as a message "Are you going for a run today?" sent by Xiao Ming, a contact in the social application.

[0132] After receiving an application message pushed by an application, if the phone is in the screen-off state, the screen can be briefly lit and a message notification bubble can be displayed, or the message notification bubble can be not displayed and the screen can remain unlit. As shown in Figure 7, the phone displays a black screen interface. If the phone is in the lock screen state, the message notification bubble can be briefly displayed, or the message notification bubble can be not displayed and the lock screen interface can be continued. It should be noted that the following embodiments are described using the phone in the screen-off state as an example. The situation when the phone is in the lock screen state can be used as a reference and will not be repeated here.

[0133] Then, the mobile phone can transfer the above application message (for example, "Are you going for a run today?" pushed by the social application) to the PC through the communication channel used for transferring messages between the mobile phone and the PC.

[0134] It is understandable that the application can be installed on the PC (the corresponding computer version of the application, in this article, the corresponding mobile version and computer version of the application are collectively referred to as the application, without distinguishing them in name) or not. In other words, whether the corresponding application is installed on the PC does not affect the flow of messages and subsequent heterogeneous screen projection.

[0135] For example, the message card 702 displays the summary information of the application message "Are you still running today?", the device information "from the mobile phone", the application information "social application", and the contact information "Xiao Ming", so that the user can quickly and easily know the content of the application message through the message card, and the application message is a message sent by Xiao Ming to the social application on the mobile phone, which makes it easier for the user to determine whether to start screen projection to reply to the application message and other operations.

[0136] In some embodiments, after the PC displays the message card 702, the user can process the application message by operating the PC. Selecting to operate the PC to process the application message will trigger the screen projection between the mobile phone and the PC.

[0137] In other embodiments, the target device can receive one or more application messages transferred from any source device within the same trust ring. For example, in response to a user operation, the PC can display a message list window, which can centrally display received message notifications or receive and unprocessed message notifications. As an implementation, the aforementioned message notifications can also be displayed in the message list window in the form of message cards.

[0138] Take the target device as a PC with a touch screen, and the source devices as mobile phone 1 and mobile phone 2 as an example. When the PC detects a specified operation, please refer to Figure 8, which is another operation diagram for starting screen projection provided by an embodiment of the present application. When the PC detects a user's sliding operation on the touch screen, a message list window 801 can be displayed. The message list window 801 displays application messages from mobile phone 1 (such as message card 802), application messages from mobile phone 2 (such as message card 803), etc. It can be understood that the message notifications displayed in the message list window 801 are all information recently received by the PC from other devices in the trust ring. Among them, message card 802 corresponds to the message pushed by the social application in mobile phone 1. Message card 803 corresponds to the message pushed by the information application in mobile phone 2 (for example, text messages).

[0139] Similar to message card 702 displayed on the desktop, when the PC detects that the user has selected any message card in message list window 801, it can trigger the device corresponding to that message card to cast the screen. For example, when the PC detects that the user has selected message card 802, it can trigger mobile phone 1 to cast the screen. In this way, mobile phone 1 is the source device and the PC is the corresponding target device. For another example, when the PC detects that the user has selected message card 803, it can trigger mobile phone 2 to cast the screen. In this way, mobile phone 2 is the source device and the PC is the corresponding target device.

[0140] For example, the user can select the message card 702 on the PC. After the PC can detect that the user has selected the message card 702, even if the mobile phone is in the screen-off state, the PC can remotely control the mobile phone to run the social application in the background. After the social application enters the background running state, the mobile phone still displays a black screen interface. In addition, the PC can also display the projection window 703 corresponding to the social application on the desktop 701. Among them, the projection window 703 corresponding to the above-mentioned social application is used to display the application interface 704 corresponding to the application message. Among them, the application interface refers to the interface for displaying the above-mentioned application message.

[0141] It is understood that the screen projection window displaying the application interface can be referred to as the application screen projection window. Optionally, the application interface 704 includes the content of the application message, operation controls, reply box, etc. Exemplarily, the application interface 704 is a chat box interface corresponding to the application message, in which the user can view the application message and reply.

[0142] As an implementation method, after the PC determines that the user has selected the message card 702, the mobile phone can render the corresponding application interface on the virtual display area through data interaction between the PC and the mobile phone. The application interface rendered on the virtual display area is not displayed in the foreground of the mobile phone. That is to say, as shown in Figure 7, the mobile phone display still displays a black screen interface. The virtual display area is a service for the PC. When the PC calls the virtual display area, the content in the virtual display area can be displayed in the projection window 703 in the PC. In this way, when the mobile phone is in the screen-off state, the content displayed in the projection window on the PC is the application interface 704 in the virtual display area.

[0143] While the application screen is being projected, users can not only send control information to the phone through the PC, but also view the phone's response to the control information through the PC, thus achieving real-time remote control. For example, remotely input a reply on the phone and send it to a designated contact (Xiao Ming). Another example is to remotely make and receive audio and video calls, copy, cut, delete, etc.

[0144] It should also be noted that in the scenario of using heterogeneous screen projection between the PC and the mobile phone, the display interface in the projection window 703 will not change synchronously with the interface shown on the mobile phone display. It is understandable that when the mobile phone is in the screen-off state, the mobile phone display always displays a black screen interface.

[0145] Optionally, the application interface 704 includes at least an interface element for the user to perform a reply operation. After the application interface 704 is displayed in the PC's screen projection window, the user can reply to the application message on the PC through the interface element. The application interface 704 includes a return control 707, but does not include a home button. In response to the user clicking the return control 707, the PC displays the previous application interface of the application interface 704 in the screen projection window 703. Exemplarily, in some embodiments, the previous application interface is the main interface of the application, wherein the main interface does not include a return control.

[0146] As can be seen from the above, the projection granularity of heterogeneous screen projection is application. It can be understood that in heterogeneous screen projection, the application projection window on the target device can display all application interfaces of the application, but does not display the application interfaces of other applications, including the mobile phone's desktop, so there is no need to set a return control on the main interface of the application.

[0147] Of course, a close control can be configured on the projection window 703. The PC can cancel the display of the projection window 703 in response to the user's selection of the "close control". Optionally, the projection window 703 can also include a minimize control, a maximize control, etc., which are not limited here.

[0148] Furthermore, in some embodiments, when the mobile phone is in the screen-off state and the screen projection is started from another source, the following steps may be performed after the mobile phone is unlocked:

[0149] (1) When detecting that the current display interface of the mobile phone has changed, the display content of the screen projection window 703 on the PC is maintained;

[0150] (2) After detecting that the user has performed an operation on the application interface 704, the command corresponding to the user's operation is sent to the mobile phone, and the social application running in the background of the mobile phone is operated based on the command, but the display content of the mobile phone screen is not changed;

[0151] (3) When it is detected that the user has opened a social application on the mobile phone, the PC closes the screen projection window 703 and the mobile phone display screen displays the content of the current social application.

[0152] That is, in some embodiments, when a mobile phone opens an application that is performing heterogeneous screen projection on a PC, the application will be launched to the foreground of the mobile phone and the heterogeneous screen projection will end.

[0153] In other embodiments, the source device can optionally enable a security authentication function to protect the privacy of the source device that has locked or turned off the screen. That is, when the source device (such as a mobile phone) enables the security authentication function and is configured with an unlock password, if the target device (such as a PC) determines that the source device has the conditions for screen projection, then the target device can display the screen projection window, and the screen projection window at this time is used to display the security authentication interface. Among them, the above-mentioned unlock password can be an authentication password used to release the lock screen state of the mobile phone.

[0154] It is understandable that the security authentication method can also be fingerprint authentication by the user inputting a fingerprint, voice authentication by the user inputting a voice, etc., and there is no limitation on this.

[0155] It can be understood that the projection window displaying the security authentication interface can be called the security authentication window.

[0156] Continuing with Figure 7, after the PC detects that the user has selected message card 702, if it determines that the phone is ready for screen projection, the PC may display screen projection window 703. Optionally, the screen projection window 703 may first display a security authentication interface 705. Thus, after the PC receives the phone unlock password entered by the user from security authentication interface 705 and the unlock password is verified, the PC may control screen projection window 703 to switch to display a transitional loading interface 706 indicating "Waiting Messages."

[0157] It can be understood that the projection window that displays the "prompt waiting information" transition loading interface can be called a transition loading window.

[0158] While the projection window 703 displays the "Waiting Information" transition loading interface 706, the mobile phone responds to the PC's instructions, starts the social application, draws the application interface of the social application in the virtual display area, and obtains the drawing parameters of each display object in the virtual display area. Among them, the above-mentioned virtual display area is a service for the PC, which is not visible on the display screen of the mobile phone. The above-mentioned drawing parameters are related parameters for rendering the interface, which can also be called display data in some examples.

[0159] After the mobile phone completes drawing the above application interface, it sends the corresponding display data to the PC, instructing the PC to render the above application interface in the projection window 703. In this way, the projection window 703 switches to display the application interface 704 of the social application. In other words, the mobile phone projects the contents of the virtual display area to the PC's projection window 703. In other words, after the PC receives the mobile phone unlock password entered by the user and the unlock password is verified, the mobile phone begins to cast the screen to the PC.

[0160] It is understood that the terms "interface" and "application interface" in the specification, claims, and drawings of this application refer to the media interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. A common form of expression is the graphical user interface (GUI), which refers to a user interface related to computer operations displayed in a graphical manner. It can be an interface element such as an icon, window, or control displayed on the display screen of an electronic device, where a control can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0161] It can be understood that in the scenario where the above-mentioned source device is in a locked screen or off screen state, according to a screen projection method provided by this embodiment, the user can directly operate on the target device to realize the projection of the application window corresponding to the flow message on the target device. Based on the projection of the application window on the target device, the function of multi-screen collaborative interaction across devices is realized through the coordination of control and input. In this way, the user can focus on the operation of the target device without having to operate the source device in the locked screen or off screen state, which reduces the user operation of starting the projection, improves the efficiency of human-computer interaction in starting the projection, and enhances the user experience.

[0162] For example, when the mobile phone is in the screen-off or locked state, there is such a scenario: when the user is using a PC to play games or work in the study or living room at home, the mobile phone is placed in another room; when a contact (Xiao Ming) on ​​a social application sends a message to the user's mobile phone, the user can immediately know on the PC that his mobile phone has received a message notification. However, in this scenario, the user is often focused on playing games or working on the PC, and does not want to get up and go to the room to get the mobile phone to check or reply to messages, but wants to start the application window displaying the message on the PC (that is, the screen projection window corresponding to the mobile phone) and process the message through the PC. However, since the mobile phone in the other room is in the lock screen state at this time, based on the existing technology, the user cannot directly start the application window displaying the message on the PC. The screen projection window must first be unlocked in the room and put in the bright screen state before the corresponding screen projection window of the mobile phone can be started on the PC.

[0163] By adopting the screen projection method provided in this application, even in the face of the aforementioned scenario, the user no longer needs to go to other rooms to operate the mobile phone, but can continue to stay in the study or living room. By simply clicking on the message card corresponding to the mobile phone on the PC, the screen projection window corresponding to the mobile phone can be started on the PC to realize the window display of heterogeneous screen projection, thereby facilitating the user to process the message and improving the user experience.

[0164] In other embodiments, the target device can receive one or more application messages transferred from any source device within the same trust ring. For example, in response to a user operation, the PC can display a message list window, which can centrally display received message notifications or receive and unprocessed message notifications. As an implementation, the aforementioned message notifications can also be displayed in the message list window in the form of message cards.

[0165] Take the target device as a PC with a touch screen, and the source devices as mobile phone 1 and mobile phone 2 as an example. When the PC detects a specified operation, please refer to Figure 8, which is a schematic diagram of displaying a message list window. The PC detects a user's sliding operation on the touch screen and can display a message list window 801. The message list window 801 displays application messages from mobile phone 1 (such as message card 802), application messages from mobile phone 2 (such as message card 803), etc. It can be understood that the message notifications displayed in the message list window 801 are all information recently received by the PC from other devices in the trust ring. Among them, message card 802 corresponds to the message pushed by the social application in mobile phone 1. Message card 803 corresponds to the message pushed by the information application in mobile phone 2 (for example, text messages).

[0166] Similar to message card 702 displayed on the desktop, when the PC detects that the user has selected any message card in message list window 801, it can trigger the device corresponding to that message card to cast the screen. For example, when the PC detects that the user has selected message card 802, it can trigger mobile phone 1 to cast the screen. In this way, mobile phone 1 is the source device and the PC is the corresponding target device. For another example, when the PC detects that the user has selected message card 803, it can trigger mobile phone 2 to cast the screen. In this way, mobile phone 2 is the source device and the PC is the corresponding target device.

[0167] It is understood that the embodiments of the present application provide a method for screen projection that can be applied to electronic devices such as mobile phones, tablet computers, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The embodiments of the present application do not impose any restrictions on the specific type of electronic device.

[0168] For example, please refer to FIG9 , which is a schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of the present application.

[0169] The electronic device 100 may include a processor 110, an antenna 1, an antenna 2, a mobile communication module 120, a wireless communication module 130, an audio module 140, a speaker 140A, a receiver 140B, a microphone 140C, an earphone jack 140D, a sensor module 150, a display screen 160, etc. The sensor module 150 may include a proximity light sensor 150A, a fingerprint sensor 150B, a touch sensor 150C, an ambient light sensor 150D, etc.

[0170] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0171] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0172] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 120, the wireless communication module 130, the modem processor and the baseband processor.

[0173] The mobile communication module 120 can provide wireless communication solutions including 2G / 3G / 4G / 5G applied on the electronic device 100.

[0174] The wireless communication module 130 can provide wireless communication solutions for application on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0175] Electronic device 100 implements display functionality through a GPU, display screen 160, and an application processor. A GPU is a microprocessor for image processing that connects display screen 160 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0176] The electronic device 100 can implement audio functions, such as voice calls, through the audio module 140 and the application processor.

[0177] The speaker 140A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 140A.

[0178] The receiver 140B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 140B close to the ear to hear the voice.

[0179] Microphone 140C, also known as a "microphone" or "speaker," is used to convert sound signals into electrical signals. When making a call or sending a voice message, a user can speak by placing their mouth close to microphone 140C, inputting the sound signal into microphone 140C.

[0180] The headphone jack 140D is used to connect a wired headphone.

[0181] The proximity light sensor 150A may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The electronic device 100 can use the proximity light sensor 150A to detect when a user holds the electronic device 100 close to their ear to make a call, automatically turning off the screen to save power. The proximity light sensor 150A can also be used to automatically unlock and lock the screen in both holster and pocket modes.

[0182] The fingerprint sensor 150B is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0183] Touch sensor 150C, also known as a "touch device," can be disposed on display screen 160. Touch sensor 150C and display screen 160 form a touch screen, also known as a "touch screen." Touch sensor 150D is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operation to an application processor to determine the type of touch event. Visual output related to the touch operation can be provided via display screen 160. In other embodiments, touch sensor 150C can also be disposed on the surface of electronic device 100, at a location different from that of display screen 160.

[0184] Ambient light sensor 150D is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 160 based on the perceived ambient light brightness. Ambient light sensor 150D can also be used to automatically adjust white balance when taking photos. Ambient light sensor 150B can also work with proximity light sensor 150A to detect whether electronic device 100 is in a pocket to prevent accidental touches.

[0185] The following describes a system structure of an electronic device provided by an embodiment of the present application in conjunction with FIG10 .

[0186] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. It should be noted that the right half of Figure 10 shows the software architecture of a source device, such as a mobile phone. The left half of Figure 10 shows the software architecture of a target device, such as a tablet or PC.

[0187] In some embodiments, as shown in the right half of Figure 10, the software framework of the source device may include an application layer, a capability service layer, an application framework layer (framework, FWK), an Android runtime (Android runtime) and a system library and driver layer.

[0188] (1) Application layer

[0189] The application layer may include a series of application packages, such as social, information, call, camera and other applications (also referred to as applications). For example, social applications include WeChat, QQ, etc., and information applications include text messages, etc. For the convenience of description, the application will be referred to as application below. The application on the source device can be a native application (such as an application installed in the source device when the operating system is installed before the source device leaves the factory), or it can be a third-party application (such as an application downloaded and installed by the user through the application store), which is not limited in the embodiments of this application.

[0190] (2) Capability service layer

[0191] The capability service layer provides capability support for implementing corresponding services. As shown in Figure 10, it may include a screen projection assistant module, a first virtualization service module, a first screen projection service module, a first device discovery authentication connection module, etc.

[0192] Among them, the screen projection assistant module can be a module used to interact with other electronic devices (such as target devices) to exchange screen projection related information, realize end-to-end logical control and connection management of screen projection, and generate a session key (SessionKey) to ensure the security of the communication session between the source device and the target device. The screen projection assistant module may include functional modules such as a screen projection management module, a heterogeneous switching module, and a virtualization service initialization module. The screen projection management module is responsible for managing screen projection related matters, such as adding and removing related processing logic settings of VirtualDisplay. Among them, VirtualDisplay can also be called a virtual display area, virtual screen or virtual Display, etc. The heterogeneous switching module is used to control the source device to switch from a heterogeneous screen projection mode to a homogeneous screen projection mode. The virtualization service initialization module makes initialization settings for the virtualization service that starts heterogeneous screen projection.

[0193] The first virtualization service module is used to implement end-to-end logical control of audio and video stream projection, and is responsible for the audio and video data streams generated by speakers, microphones, cameras, etc. transmitted between the source device and the target device.

[0194] The first screen projection service module may include a first display virtualization module and a first reverse control module, which respectively provide screen projection capabilities (i.e., the content of the source device is projected to the target device for display) and reverse events (i.e., events that control the source device triggered by the target device) basic control capabilities. The first screen projection service module receives instructions from the screen projection management module and provides corresponding screen projection services according to the instructions, thereby realizing the projection of programs running on the local machine to other electronic devices (such as the target device). The first screen projection service module injects reverse control and input method into the display of the target device (hereinafter referred to as the external screen), and adds a virtual display tag (such as Flag1) when creating a virtual display area, so that the external screen only displays its own content, and does not mirror the content of the display of the source device (hereinafter referred to as the main screen), thereby realizing heterogeneous screen projection.

[0195] The first device discovery connection authentication module is responsible for discovery, connection, authentication and other tasks, and is used to control the driver layer to implement proximity discovery, authentication, connection and other functions between the source device and the target device, and may include a first communication module and a first security authentication module. The first communication module provides a signaling transmission channel between the source device and the target device, and is used to send instructions to the driver layer to establish a communication connection and a screen projection connection between the source device and the target device. At the same time, the first communication module is also used to start the screen projection assistant module after the transmission channel is established. The first security authentication module is used for security authentication, and is responsible for identifying the owner and starting distributed unlocking authentication, obtaining a trust ring, and ensuring the privacy and security of the user.

[0196] (3) Application Framework Layer (FWK)

[0197] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. As shown in Figure 10, the application framework layer can include the Activity Manager Service (AMS), the Window Manager Service (WMS), the multi-screen framework module, the System UI module, and so on.

[0198] The AMS is responsible for managing activities, starting, switching, and scheduling various components in the system, as well as managing and scheduling applications. For each activity, AMS has a corresponding application record (ActivityRecord), which records the status of the application's activities. Each activity creates a corresponding ActivityRecord object in AMS, which AMS uses as an identifier to schedule the application's activity processes. These ActivityRecord objects are managed in their respective task stacks, and each task stack can have multiple ActivityRecord objects. All task stacks are centrally managed by the ActivityStack, which is responsible for the order, push, and pop of TaskStacks. Specifically, AMS defines data classes for storing processes, activities, and tasks. The data class corresponding to a process can include process file information, memory status information, and the activities and services contained in the process. Activity information can be stored in the ActivityStack, which is used to centrally schedule application activities. The ActivityStack specifically stores information about all currently running Activities (i.e., the final ArrayList mHistory), such as interface configuration information. For example, the currently running Activity can be stored in the newArrayList. The ActivityStack also stores information about previously run Activities, such as interface configuration information. It's important to note that an Activity does not correspond to a single application; only an ActivityThread corresponds to an application. Therefore, Android allows multiple applications to run simultaneously, which actually means it allows multiple ActivityThreads to run simultaneously. The basic concept of activity scheduling in Android is this: each application process reports to the Active Directory Management Service (AMS) when it wants to start a new Activity or stop the current Activity. The AMS internally keeps a record of all application processes. When the AMS receives a start or stop report, it first updates the internal record and then notifies the corresponding client process to run or stop the specified Activity. Because the AMS maintains a record of all Activities, it can schedule these Activities and automatically close background Activities based on the status of the Activities and system memory.

[0199] WMS carries data and attributes related to the "interface" and is used to manage and control the status related to the "interface", that is, to manage the graphical user interface (GUI) resources used on the mobile phone screen, such as managing window programs and event dispatching. Among them, managing window programs refers to outputting to the physical screen or other display devices in an orderly manner with the assistance of the application server and WMS according to the display request of the application. Event dispatching refers to dispatching user events from the keyboard, physical buttons, touch screen, mouse, trackball (TraceBoll), etc. to the corresponding controls or windows. The window management program can also obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc., and is responsible for managing the display mode of the application window, including the coordinate size of the window display, the window display level, etc. Specifically, it may include: creating and destroying windows, showing and hiding windows, window layout, focus management, and input method and wallpaper management, etc. In some embodiments of the present application, WMS can ensure the visibility of the projection window when the mobile phone is in the locked screen and screen off state.

[0200] It should be noted that the task stack management function of AMS is used to manage the life cycle of Activities and the scheduling between Activities. The life cycle management function of WMS is used to manage the life cycle of windows. The two can cooperate with each other to activate the window through the callback (onResume) interface, and the activated window is displayed in the foreground; or, the window can be paused through the pause (onPause) interface to switch the window from the foreground to the background. Taking the topmost window (top Activity) in the task stack as an example, the top Activity calls onResume to activate it, then the top Activity is displayed in the foreground. When a new window is pushed into the task stack, and the new window is located on the upper layer of the original top Activity, the original top Activity calls onPause to pause the window to switch the original top Activity from the foreground to the background. A similar principle can be used to manage the life cycle of the new window.

[0201] The multi-screen framework module is responsible for implementing the heterogeneous screen projection logic, calling the PendingIntetRecord interface to start the Activity corresponding to the PendingIntet (specifying the send behavior), and launching the application corresponding to the Activity into the virtual display area by adding a heterogeneous screen projection tag (e.g., Flag2). PendingIntent is a capability provided by Android for external programs to call their own programs. Its lifecycle is independent of the main program, and it executes specified behaviors when certain conditions are met or certain events are triggered. The multi-screen framework module is also used to monitor sensor events and interact with the screen projection assistant module. For example, it calls ActivityRecord to monitor whether the Activity is in the foreground and visible, and notifies the screen projection assistant module of the monitoring results; it calls Task to monitor whether the virtual display area stack is empty and notifies the screen projection assistant module of the monitoring results. The multi-screen framework module can be used to translate and encapsulate the original input events, obtain input events containing more information, and send them to the WMS, which stores the clickable areas (such as controls) of each application and the location information of the focus window. Therefore, the WMS can correctly distribute input events to the specified control or focus window. The multi-screen framework module can respond to the received input events and forward the input events to the WMS. The WMS distributes them to the corresponding applications. The applications send registration requests to the AMS and the applications are started.

[0202] In addition, the multi-screen framework module is also used to switch the application projected to the external screen back to the main screen when the user clicks the recent task card or desktop icon. At this time, if the application projected to the external screen adopts a heterogeneous screen projection method, when switching the application back to the main screen, it is necessary to call the heterogeneous switching module mentioned above to switch the heterogeneous screen projection method to the homogeneous screen projection method, otherwise the application cannot be displayed on the main screen. The multi-screen framework module is also used to remove the security layer when the source device is in the locked screen state. In some embodiments, the multi-screen framework module can also be used to make relevant modifications to the power button of the mobile phone in the screen projection scenario.

[0203] The SystemUI module is a set of UI components that provide users with system-level information display and interaction. It is mainly used to implement status bar information display (such as battery, Wi-Fi signal, 3G / 4G icon display), notification panel (such as system messages and third-party message notifications), recent taskbar display panel (such as displaying recently used applications), etc. In the embodiment of the present application, the SystemUI module is responsible for notification services, which is used to transfer message notifications from the source device to the target device.

[0204] It is understood that the application framework layer may also include a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. (not shown in FIG10 ). The specific meanings thereof may be referred to in relevant technical documents and will not be described here.

[0205] (4) Android runtime and system libraries

[0206] The Android runtime is responsible for system scheduling and management. It consists of core libraries and a virtual machine. The core libraries consist of two parts: one containing the functional functions called by programming languages ​​(e.g., Java) and the other containing the system's core libraries. The application layer and application framework layer run in the virtual machine. The virtual machine executes the application layer and application framework layer programming files (e.g., Java files) as binary files. The virtual machine performs functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0207] The system library can include multiple functional modules. For example: Surface Manager, Media Libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc. The surface manager is used to manage the display subsystem and provides the fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications. The media library supports the playback and recording of various common audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0208] (5) Driver layer

[0209] The driver layer is the foundation of the Android operating system, and all of the Android operating system's ultimate functions are completed through the driver layer. As shown in Figure 10, the driver layer includes low-level drivers, responsible for tasks such as discovery, authentication, and connection. For example, the driver layer receives commands from the communication module in the application framework layer and executes actions such as connection and disconnection. Specifically, the driver layer includes Wi-Fi P2P chips and Bluetooth chips, which can establish Wi-Fi P2P and Bluetooth transmission channels. Of course, the driver layer can also include hardware drivers such as display drivers, camera drivers, audio drivers, sensor drivers, and virtual card drivers.

[0210] It should be noted that the software structure diagram of the electronic device shown in Figure 10 provided in this application is only an example and does not limit the specific module division in different layers of the Android operating system. For details, please refer to the introduction of the Android operating system software structure in conventional technology. In addition, the screen projection method and device provided in this application can also be implemented based on other operating systems, and this application will not give examples one by one.

[0211] In some embodiments, as shown in the left half of Figure 10, the software framework of the target device may include a PC manager, an operating system, and a driver layer. Some functional modules of the target device and their functions can refer to the above-mentioned source device and will not be repeated here.

[0212] Specifically, the PC manager of the target device includes a message center module, a second virtualization service module, a second screen projection service module, and a second device discovery connection authentication module.

[0213] The Message Center module is used to interact with messages and signaling from the source device, receiving message notifications from the source device and initiating session instructions (openSession) to the source device. It is also used to issue instructions to start the screen projection service after a session is established.

[0214] The functions of the second virtualization service module, the second screen projection service module, and the second device discovery authentication connection module can respectively refer to the first virtualization service module, the first screen projection service module, and the first device discovery connection authentication module in the above-mentioned source device. The difference is that the second screen projection service module is used to receive and parse video streams and related information sent by other devices, so as to display applications running on other devices on the local device.

[0215] The operating system of the target device can be Windows OS (such as a personal computer) or Android OS (such as a tablet).

[0216] For the driver layer of the target device, please refer to the above source device.

[0217] Based on the system structure of the electronic device shown in FIG10 , taking a mobile phone as a source device and a PC as a target device as an example, the specific implementation of the aforementioned embodiment is described in conjunction with FIG11 to FIG13 .

[0218] Please refer to Figures 11 to 13, which are timing diagrams of a method and device for screen projection provided in an embodiment of the present application. It can be understood that Figures 11 to 13 are a set of continuous timing diagrams, which illustrate the interaction process between the various functional modules shown in Figure 11.

[0219] For example, the mobile phone side includes the SystemUI module, the first communication module, the screen projection assistant module, the first virtualization service module, the first screen projection service module, the first security authentication module, FWK, and the application; the PC side includes the message center module, the second screen projection module, the second communication module, and the second security authentication module. It should be noted that the other functional modules shown in Figure 10 also participate in the specific implementation process of the screen projection method and device provided in the embodiment of the present application, and are not shown in this timing diagram.

[0220] It should be noted that, in the embodiments of this application, unless otherwise specified, the mobile phone is in a locked screen or off screen state.

[0221] 1. The source device and the target device establish a first communication connection. The application message flow of the source device is transferred to the target device, and a message notification control is displayed on the target device (see Figure 11).

[0222] S101: The message center module of the PC receives an operation of selecting a mobile phone by a user.

[0223] It is understandable that S101 is only an example of establishing a first communication connection between a PC and a mobile phone. The PC detects a device in the same local area network as the PC, displays all connectable devices for the user to select, and the user selects the mobile phone on the PC to initiate subsequent connection behavior. There are other ways to initiate the first communication connection between the PC and the mobile phone. For example, the mobile phone detects a device in the same local area network as the PC, displays all connectable devices for the user to select, and the user selects the PC on the mobile phone to initiate subsequent connection behavior. For another example, the mobile phone and the PC can also establish a near-field connection through the "touch and pay" function, the NFC function, etc., which is not limited here.

[0224] S102: The message center module sends a message to the second communication module instructing to establish a communication connection with the mobile phone. The message carries the device identification code of the mobile phone.

[0225] The user selects a mobile phone as a collaborative notification device from among the available collaborative notification devices found by the PC. After receiving the selection operation input by the user, the message center module of the PC can send an inter-process communication (IPC) message to the second communication module. Optionally, the IPC message can carry the device name, device identification code, MAC address, etc. of the mobile phone, which is used to instruct the second communication module to establish a communication connection with the mobile phone.

[0226] S103: The second communication module establishes a first communication connection with the first communication module of the mobile phone.

[0227] Exemplarily, the first communication connection may be a Bluetooth connection (BR connection), which is used to implement collaborative notifications between the PC and the mobile phone, that is, messages on the mobile phone can be transferred to the PC through the Bluetooth channel.

[0228] S104: The second communication module returns a message to the message center module indicating that the first communication connection is successfully established.

[0229] S105: The application of the mobile phone receives an application message.

[0230] After the PC and the mobile phone establish a first communication connection, the mobile phone receives an application message pushed by the application installed on the mobile phone. The application message can have different forms of expression in the application on the mobile phone (such as the first data), the SystemUI module, and the message center module on the PC (such as the first message), collectively referred to here as application messages.

[0231] S106: The application sends an application message to the SystemUI module. The application message carries the message content and the application package name.

[0232] S107: The SystemUI module instructs the first communication module to forward the application message to the PC.

[0233] For example, in an embodiment of the present application, after the PC and the mobile phone establish a first communication connection, the SystemUI module will transfer all messages received by the mobile phone to the PC based on the first communication connection; it can also be transferred selectively, for example, messages with a special tag will be transferred to the PC, or messages from specific applications will be transferred to the PC, or messages judged to be of higher importance will be transferred to the PC, etc., which are not limited here.

[0234] S108: The first communication module sends an application message to the second communication module based on the first communication connection. The application message carries message content and an application package name.

[0235] S109: The second communication module sends an application message to the message center module. The application message carries the message content and the application package name.

[0236] S110: The message center module receives the application message and controls the display screen to display the corresponding message notification control.

[0237] After the PC and mobile phone establish a first communication connection, application messages received on the mobile phone are transferred to the PC. It is understood that whether the mobile phone is unlocked and the screen is on, locked, or off does not affect the flow of message notifications between the PC and the mobile phone. After the PC receives the application message, it displays the message notification control associated with the application message.

[0238] For example, please refer back to FIG7 . When the mobile phone is in the screen-off state and receives an application message, a message notification control associated with the application message, such as a message card 702 , is displayed on the PC.

[0239] 2. When the source device is in the locked or off state, the target device receives a user operation on the message notification control, and the target device establishes a third communication connection with the source device. The target device sends a request to start heterogeneous screen casting to the source device. When the source device is in the locked or off state, the application screen casting window is displayed on the target device (see Figures 12-13)

[0240] S201: When the mobile phone is in a locked screen or off screen state, the message center module receives a first operation of the user clicking a message notification control.

[0241] S202: The message center module sends a message to the second communication module instructing to create a second session. The message carries the service channel name (WI-FI P2P channel) and the device identification code of the mobile phone.

[0242] Among them, exemplarily, the service channel name can be a WI-FI P2P channel, the device identification code is the identification code of the mobile phone, and the above-mentioned instruction to create a second session is used to instruct the second communication module to establish a WI-FI P2P channel between the PC and the mobile phone corresponding to the device identification code.

[0243] S203: The second communication module establishes a third communication connection with the first communication module.

[0244] Exemplarily, the third communication connection may be a WI-FI P2P connection, providing a data transmission channel for the screen projection service.

[0245] It should be noted that the types of the first communication connection, the second communication connection, and the third communication connection established between the PC and the mobile phone are not limited to Bluetooth connection and Wi-Fi P2P connection, and can also be other types of connection methods as long as they can meet the requirements for data transmission between the two.

[0246] S204: The first communication module sends a message instructing the screen projection assistant module to start.

[0247] S205: The screen projection assistant module is started.

[0248] The first communication module is also responsible for interacting with the screen projection assistant. The screen projection assistant module is in a closed state when the mobile phone has no screen projection demand. The first communication module triggers the screen projection assistant module to start when there is a screen projection demand.

[0249] S206: The second communication module returns a message to the message center module indicating that the third communication connection is successfully established.

[0250] S207: Based on the third communication connection, the message center module negotiates parameters with the screen projection assistant to determine the session key and the IP address of the other party.

[0251] Parameter negotiation involves negotiating a SessionKey on the service side and obtaining the peer device's IP address. The SessionKey, also known as a data encryption key or working key, is a randomly generated encryption and decryption key used to ensure secure communication between a user and another computer, or between two computers. It is negotiated between communicating users and is typically generated dynamically, only when session data encryption is required.

[0252] S208: The message center module sends a message indicating startup to the second screen projection service module, which carries the session key and the IP address of the mobile phone.

[0253] S209: The second screen projection service module is started and initialized according to the session key and the IP address of the mobile phone.

[0254] S210: The second screen projection service module returns a message indicating that initialization is completed to the message center module.

[0255] As can be seen from the above, the screen projection service module provides screen projection capabilities and basic reverse event control capabilities. After the screen projection service on the PC side is successfully initialized, it provides the ability to receive and parse the relevant information sent by the mobile phone side, so that the application running on the mobile phone can be displayed on the PC side. At the same time, the basic reverse event control capability provides support for heterogeneous screen projection. Therefore, before performing heterogeneous screen projection services, the screen projection service module needs to be initialized to ensure the normal provision of screen projection capabilities and basic reverse event control capabilities.

[0256] S211: The message center module sends a message to the second screen projection service module instructing to start the screen projection service.

[0257] S212: Based on the third communication connection, the second screen projection service module sends a message to the screen projection assistant module to start the screen projection service.

[0258] It should be noted that, as shown in the figure, step S207 includes the second screen projection service module sending a message to the second communication module, the second communication module then transmits the message to the first communication module through the third communication connection, and the first communication module then sends the message to the screen projection assistant module. For the convenience of description, the following "based on the third communication connection" includes the data transmission process between the first communication module and the second communication module, which will not be repeated below.

[0259] S213: The screen projection assistant module is bound to the screen projection service.

[0260] It can be understood that the screen projection management module in the screen projection assistant module is bound to the relevant screen projection services to manage screen projection affairs.

[0261] S214: The screen projection assistant module sends an initialization instruction message to the first virtualization service module.

[0262] It can be understood that the virtualization service initialization module in the screen projection assistant module is used to instruct the virtualization service module to initialize.

[0263] S215: The first virtualization service module completes initialization.

[0264] The first virtualization service module is responsible for the audio and video virtualization in the screen projection service. The interaction between the first virtualization service module on the mobile phone side and the second virtualization service module on the PC side (not shown in Figure 13) can enable the mobile phone application projected to the PC to use the PC's speaker, microphone, and camera during the screen projection process, and transmit the generated audio and video data streams between the mobile phone and the PC, thereby realizing functions such as making and receiving audio and video calls during screen projection.

[0265] S216: The first virtualization service module sends a message to the first screen projection service module to instruct the start of the screen projection service.

[0266] S217: The first screen projection service module is started.

[0267] S218: The first screen projection service module creates a virtual display and adds a virtual display tag to it.

[0268] It should be noted that the screen projection service module is an important functional module for implementing heterogeneous screen projection. The specific implementation method is that the first screen projection service module on the mobile phone adds Flag1: VIRTUAL_DISPLAY_FLAG_OWN_CONTENT_ONLY (virtual display tag) when creating the virtual display area. This is used to ensure that the screen projection window on the PC only displays its own content, and does not mirror the content displayed on the mobile phone display. This ensures that the display content of the mobile phone does not affect the display content of the PC screen projection window.

[0269] This can achieve two purposes: first, when the phone is locked or in the off state, the phone displays a black screen or a lock screen interface, and the PC's screen projection window will not display a black screen or a lock screen interface; second, when the phone is in use, the phone displays other application interfaces or desktops, which will not affect the display of the PC's screen projection window. The PC's screen projection window can maintain its own content without being synchronized with the phone's display content. At the same time, the reverse event control basic capability provided by the screen projection service module can achieve reverse control of the mobile phone end on the computer side. Of course, the implementation of the heterogeneous screen projection method also requires the second screen projection service module on the PC side to interact with the first screen projection service module on the mobile phone side, which is not shown in Figure 12.

[0270] It should be noted that if the phone is in the locked or off state, even if the application layer corresponding to the currently executed Activity has been drawn and rendered to the projection window, the Android system will, by default, overlay the lock screen layer on the phone's display over the VirtualDisplay. This means that the projection window on the PC display will also be covered by the lock screen layer, that is, the projection window on the PC will also display the lock screen layer, resulting in abnormal projection. However, adding Flag1 to the VR area will not mirror the content displayed on the phone's display to the PC's projection window, achieving heterogeneous projection. This solves the problem of the projection window displaying a black screen or lock screen interface as shown in Figures 5 and 6.

[0271] S219: The first screen projection service module sends a message to the screen projection assistant module indicating that the screen projection service has been started.

[0272] S220: Based on the third communication connection, the first screen projection assistant sends a message to the message center module indicating that the screen projection service has been started.

[0273] S221: Based on the third communication connection, the message center module sends a message to SystemUI to start the application for screen projection, carrying the application package name.

[0274] S222: SystemUI sends a message to FWK to start application screen projection, and the message carries the application package name.

[0275] S223: FWK starts application screen projection.

[0276] It should be noted that when the phone's screen is locked or off, in order to ensure that the heterogeneous screen projection window on the PC can display the application interface normally, FWK is required to control the visibility of the application's heterogeneous screen projection window. It is understood that the specific implementation process of FWK involves functional interaction between FWK's embedded classes. The interaction process is described below and is not detailed here.

[0277] S224: FWK sends the display data of the application screen projection to the first screen projection service module.

[0278] S225: Based on the third communication connection, the first screen projection service module sends the display data of the application projection to the second screen projection service module.

[0279] S226: The second screen projection service module controls the display screen to render the above display data to the screen projection window.

[0280] S227: The second screen projection service module sends a message to the message center indicating that the rendering of the application screen projection interface of the projection window is completed.

[0281] S228: The message center module controls the display screen to display the application projection window.

[0282] At this point, the PC displays the application screen projection window corresponding to the application message, and the screen projection is successful when the phone is in the locked screen state.

[0283] For example, referring back to Figure 7 , when the phone is in the screen-off state and the PC receives a user click on a message card 702, the PC displays a projection window 703 corresponding to the social application, which is used to display the application interface 704 corresponding to the application message. The application projection window can be projection window 703 for displaying application interface 704.

[0284] The following describes the specific implementation process of FWK starting application screen projection in the aforementioned embodiment with reference to FIG14 .

[0285] When the phone is locked or off, the activity corresponding to the app message will enter a dormant state. Its lifecycle will transition from the OnResume state (visible and interactive to the user) to the OnPause state (inactive). Therefore, the app interface will not display properly when projecting the screen to a PC. To achieve app projection, it is necessary to ensure that the app's activity skips the dormant state and that the currently executing activity remains in the OnResume state to ensure the visibility of the corresponding projection window.

[0286] Please refer to Figure 14, which shows the specific implementation process of step S223 in Figure 13. As shown in Figure 14, FWK also includes embedded classes such as PendingIntentRecord (asynchronous processor), ActivityStarter (activity starter), ActivityTaskManagerServiceEx (activity stack management server), ActivityTaskSupervisor (activity stack manager), KeyguardController (lock screen controller), ActivityRecord (activity recorder), Task (task stack), RootWindowsContainer (root window container), and SurfaceFlinger (layer compositor).

[0287] It can be understood that steps S301 to S318 shown in FIG14 are between steps S222 and S224 shown in FIG13 .

[0288] S222: The SystemUI module sends a request to PendingIntentRecord to start application screen projection, and the request carries the application package name (SystemUI sends a message to FWK to start application screen projection, and the message carries the application package name).

[0289] S301: PendingIntentRecord marks all the applied Activities with a projection tag.

[0290] It should be noted that only the Activity with the projection tag added will be started for projection. Step S301 is equivalent to an Activity filter, which filters out all the Activities of the application, thereby marking all the Activities of the application that need to start projection with special marks.

[0291] For example, application message 1 corresponds to Activity 1, and PendingIntentRecord marks Activity 1 with the heterogeneous casting tag Flag2: Intent.java--->FLAG_HW_ACTIVITY_PC_CAST_NOTIFICATION=0x04000000. The application can also include Activity 2 and Activity 3, which are also marked with the heterogeneous casting tag Flag2. Because after the application is cast, the user can control and execute other activities of the application in the application casting window, so all activities of the application need to be filtered out to perform the following related steps.

[0292] S302: PendingIntentRecord sends a request to ActivityStarter to start the Activity marked with the screen projection tag, and the request carries the application package name.

[0293] S303: ActivityStarter sends a request to ActivityTaskSupervisor to start application screen projection.

[0294] S304: ActivityTaskSupervisor sends a message to KeyguardController instructing to set the Activity with the projection tag to the OnResume state.

[0295] It should be noted that ActivityTaskSupervisor listens to whether the Activity is in the OnResume state by calling KeyguardController. If so, it means that the application projection window corresponding to the Activity is visible to the user and can interact with the user after it is launched on the PC (OnResume state); if the Activity is in the stopped state (OnPause state), it means that the application projection window corresponding to the Activity is in the background after it is launched on the PC, and the user cannot operate it. Among them, the OnResume state and the OnPause state are two states in the life cycle of the Activity. When the Activity is in the OnResume state, the Activity is in an activated state that is visible and can interact with the user, and the user can get the focus of the activity; when the Activity is in the OnPause state, the Activity is covered by another transparent or Dialog-style Activity. At this time, it still maintains a connection with the window manager, and the system continues to maintain its internal state. It is still visible, but it has lost focus and cannot interact with the user. When the phone is in the locked or off screen state, the AMS component will switch the Activity lifecycle to the onPause state during the process of starting the Activity, causing the Activity to be unable to interact with the user, resulting in screen projection exceptions.

[0296] S305: KeyguardController identifies the Activity with the projection tag and sets it to the OnResume state.

[0297] KeyguardController is a lock screen controller that identifies an Activity with the cast label and returns "true" to indicate that the Activity is set to the OnResume state, ensuring that when the source device is in the lock screen state, the application cast window corresponding to the Activity with the cast label is visible on the target device.

[0298] S306: KeyguardController sends a message to ActivityRecord instructing the Activity with the projection tag to keep the OnResume state.

[0299] S307: ActivityRecord controls the Activity with the projection tag to maintain the OnResume state.

[0300] S308: ActivityRecord sends a verification message to Task, instructing Task to verify the status of the Activity at the top of the task stack.

[0301] S309: Task determines that the Activity at the top of the task stack is in the OnResume state.

[0302] It's important to note that an application may correspond to one or more activities, which are stored in one or more task stacks. Task stacks follow a last-in, first-out order, with the activity at the top of the task stack being executed first. To ensure that the application projection window corresponding to the upcoming activity is visible, ensure that the activity at the top of the stack is in the OnResume state.

[0303] It can be understood that steps S308 and S309 are equivalent to a fault-tolerant mechanism, which checks whether the Activity at the top of the task stack is in the OnResume state, so these two steps are optional, that is, in other embodiments, S308 and S309 may not be included.

[0304] S310: Task instructs RootWindowsContainer to control all tasks to enter a dormant state.

[0305] S311: RootWindowsContainer traverses the activities of all tasks, skips the activities with the projection tag, and controls other activities to enter the dormant state.

[0306] It should be noted that except for the activity with the screen projection tag, which skips the hibernation process, all other activities will be put into hibernation. Therefore, RootWindowsContainer needs to traverse all activities corresponding to the task, filter out the activities with the screen projection tag, and return "false" to indicate that the activity will not enter hibernation; for other activities, it returns "true" to enter the hibernation process.

[0307] For example, Activity1 and Activity2 in the same task stack are both added with the heterogeneous screen projection tag, and RootWindowsContainer causes Activity1 and Activity2 to skip the sleep process; while Activity3 and Activity4 that are not added with the heterogeneous screen projection tag are put into sleep state.

[0308] When Activity1 is at the top of the task stack, Task checks the status of Activity1 to ensure that it is in the OnResume state. Then, after Activity1 is projected onto the screen from a different source, it is visible to the user on the PC and can be operated by the user in the foreground.

[0309] After executing Activity1, Activity2, which is not dormant at this time, progresses to the top of the task stack. Task verifies the status of Activity2 to ensure that it is in the OnResume state. Then, Activity2 is visible to the user on the PC after being projected from another source and can be operated by the user in the foreground. The execution process of other Activities that have been marked with projected from another source is similar and will not be repeated here. Activities that have not been marked with projected from another source will be put into dormant state by RootWindowsContainer and will not be started to display on the external screen.

[0310] S312: ActivityStarter sends a message to the application to instruct it to start.

[0311] It is understandable that when ActivityStarter sends a request to start the application screen casting in step S304, it also sends a request to start the application, because in order to achieve heterogeneous screen casting of the application, the application must be started on the mobile phone. Therefore, step S223 includes the interaction between the FWK and the application layer, that is, the ActivityStarter in the FWK calls the application layer application.

[0312] S313: Application starts.

[0313] S314: The application sends a startup success message to ActivityTaskManagerServiceEx.

[0314] S316: ActivityTaskManagerServiceEx sends a message to SurfaceFlinger instructing to start layer synthesis.

[0315] S317: SurfaceFlinger determines that the virtual display area has a virtual display tag, synthesizes the application layer on the virtual display area, and obtains corresponding display data.

[0316] SurfaceFlinger receives graphics display data from multiple sources, synthesizes them, and then sends them to the display device. For example, when you open an application, there are three common display layers: the status bar at the top, the navigation bar at the bottom or side, and the application interface. Each layer is updated and rendered separately. These interfaces are synthesized by SurfaceFlinger into a single application layer display data that is refreshed to the hardware display.

[0317] In an embodiment of the present application, SurfaceFlinger determines that the virtual display area has a virtual display tag. When synthesizing the application layer, it only obtains the layer on the virtual display area, and does not obtain the layers of other display areas to synthesize with it. In this way, the aforementioned heterogeneous screen projection method can be achieved, that is, the synthesized application layer is only related to the layer of the virtual display area. Please refer back to Figure 4. SurfaceFlinger will not obtain the layers of the visible display area, so even if the visible display area includes the lock screen interface or other application interfaces, it will not affect the interface displayed by the PC's projection window. Because SurfaceFlinger only obtains the layers of the virtual display area and passes them to the PC's projection window for display in some way.

[0318] S318: SurfaceFlinger sends display data to the first screen projection service module.

[0319] It is understood that the application layer may include the application interface corresponding to the currently executed Activity. The display data may include information such as different display objects in the application layer, the display position and size of the display objects in the virtual display screen, etc.

[0320] It should be noted that the virtual display area is the virtual display area marked Flag1 created in step S218. The content of the virtual display area is not visible to the user, and the mobile phone can use the virtual display area as a canvas that occupies a certain storage space. For example, please refer to Figure 17A, which is a process diagram for starting the first application projection window. When the application package name carried by the first request to start the projection corresponds to application 1, the PendingIntentRecord will mark all Activities of application 1 with a projection label, as shown in the task stack (created by Task) in Figure 17A. When executing Activity1 of application 1 (Activity1 corresponds to the first message), as shown in Figure 17A, the mobile phone draws the first application interface corresponding to Activity1 in the virtual display area. SurfaceFlinger determines that the virtual display area has a virtual display label, then only obtains the first application interface layer in the virtual display area, and generates the corresponding first display data and sends it to the target device.

[0321] After receiving the first display data, the target device renders the application projection interface corresponding to the first message of the first application based on the first display data and displays it in the projection window of the target device. For details, please refer to steps S225-S228 in Figure 13 and Figure 7. At this time, the projection window of the target device displays the application projection interface corresponding to the first display data.

[0322] When executing Activity2 of Application 1 (Activity2 is pushed to the top of the task stack), the phone draws the second application interface corresponding to Activity2 in the virtual display area. SurfaceFlinger determines that the virtual display area has a virtual display tag, then only obtains the second application interface layer in the virtual display area, and generates the corresponding second display data and sends it to the target device. The projection window of the target device then displays the application projection interface corresponding to the second display data, that is, the application projection interface corresponding to the first display data is switched to the application projection interface corresponding to the second display data, completing the switching of different application interfaces of Application 1 in the projection window.

[0323] For example, in some embodiments of the present application, please refer to Figure 17B, which is a process diagram for starting the second application projection window. In the projection method provided in the embodiment of the present application, when the projection window of the target device displays the application projection interface corresponding to the first application (such as, the projection window of the target device displays the application projection interface corresponding to the second display data), the source device receives a second message from the second application (also called the second data). After the second message flows to the target device, the target device displays the message notification control corresponding to the second message. After the target device receives the operation of the source device clicking the message notification control, it sends a second request to the source device to start the projection. When the source device is in the locked screen or off screen state, the first projection service module receives the second request, creates a second virtual display area (VirtualDispaly2) and adds a virtual display tag. When executing Activity3 of Application 2 (Activity3 corresponds to the second message), as shown in Figure 17B, the mobile phone draws the third application interface corresponding to Activity3 in the virtual display area. SurfaceFlinger determines that the virtual display area has a virtual display tag, then only obtains the third application interface layer in the virtual display area, and generates corresponding third display data and sends it to the target device.

[0324] After the target device receives the third display data, it draws and renders the application projection interface corresponding to the second message of the second application according to the third display data, and displays it in the projection window of the target device. For details, please refer to steps S225-S228 in Figure 13 and Figure 7. Exemplarily, the projection window of the target device switches the application projection interface corresponding to the second display data to the application projection interface corresponding to the third display data, completing the switching of the application interfaces of different applications (e.g., the first application and the second application) in the projection window.

[0325] 15 and 16 illustrate specific implementations of the aforementioned embodiments in other embodiments, when the source device has activated the security authentication function.

[0326] Please refer back to Figure 7. In other embodiments, the source device can optionally enable a security authentication function to protect the privacy of the source device with the screen locked or off. Before the screen projection window 703 displays the application interface 704, a security authentication interface 705 can also be displayed, and a transition loading interface 706 can also be displayed. In this case, the above step S201, i.e., after the message center module receives the first operation of the user clicking the message notification control, will first trigger the security authentication mechanism. For the specific implementation process, please refer to Figure 14 and the following description.

[0327] 1. When the source device is in the locked or off screen state, the target device receives a user operation on the message notification control, and the source device establishes a second communication connection with the target device (see Figure 15)

[0328] S401: When the mobile phone is in a locked screen or off screen state, the message center module receives a second operation of the user clicking a message notification control.

[0329] It should be noted that the second operation here and the first operation in step S201 in Figure 12 are both operations in which the user clicks on the message notification control. In this embodiment, since a security authentication mechanism has been added, the steps shown in Figures 15 and 16 are added before triggering the establishment of the third communication connection in Figure 12.

[0330] It is understandable that when the phone is locked or off, it does not undergo user security authentication. In this case, if the PC is allowed to directly start screen casting without passing security authentication, the user's privacy will not be guaranteed. Therefore, in some embodiments, a security authentication mechanism will be triggered before starting screen casting from the PC to ensure user privacy.

[0331] Since the embodiment of the present application directly starts screen projection on the PC, the security authentication mechanism will be triggered on the PC, and the password authentication is distributed on the mobile phone. Therefore, the second security authentication module on the PC needs to send the authentication information (password) received on the PC to the first security authentication module on the mobile phone for password authentication. The mobile phone will return the authentication result to the PC. Therefore, in order to transmit the signaling required for security authentication between the mobile phone and the PC, a transmission channel needs to be established first.

[0332] S402: The message center module sends a message to the second communication module instructing to create a first session. The message carries the service channel name (Bluetooth channel) and the device identification code of the mobile phone.

[0333] Among them, illustratively, the service channel name can be a Bluetooth channel, the device identification code is the identification code of the mobile phone, and the above-mentioned instruction to create the first session is used to instruct the second communication module to establish a Bluetooth channel between the PC and the mobile phone corresponding to the device identification code.

[0334] S403: The second communication module establishes a second communication connection (Bluetooth connection) with the first communication module.

[0335] Exemplarily, the second communication connection may be a Bluetooth connection (BR connection), which provides a securely authenticated signaling transmission channel.

[0336] S404: The first communication module sends a message to the screen projection assistant to instruct it to start.

[0337] S405: The screen projection assistant starts.

[0338] S406: The second communication module returns a message to the message center module indicating that the second communication connection is successfully established.

[0339] S407: The message center module sends a message instructing the second screen projection service module to start.

[0340] S408: The second screen projection service module is started.

[0341] S409: The second screen projection service module returns a message indicating successful startup to the message center module.

[0342] It should be noted that at this time, the screen projection assistant on the mobile phone and the second screen projection service module on the PC have been started, providing the necessary functions for the subsequent screen projection service.

[0343] 2. The target device sends a security authentication request to the source device, and the source device returns a successful authentication result (see Figure 16)

[0344] S501: The message center module sends a security authentication request to the second security authentication module of the PC.

[0345] S502: The second security authentication module controls the display screen of the PC to display a security authentication window.

[0346] For example, please refer back to the security authentication window shown in FIG7 , that is, the security authentication interface 705 is displayed in the screen projection window 703 , and the user can enter the password in the above security authentication window.

[0347] It is understood that security authentication methods are not limited to password input for security authentication. For example, in the fingerprint unlocking scenario, fingerprints are collected for verification; in the voice unlocking scenario, voices are collected for verification. This is not limited here, and password unlocking is only used as an example.

[0348] S503: The second security authentication module receives the password input by the user through the security authentication window.

[0349] S504: The second security authentication module sends the password to the second communication module.

[0350] S505: The second communication module sends the password to the first communication module based on the second communication connection.

[0351] S506: The first communication module sends the password to the first security authentication module of the mobile phone.

[0352] It is understood that the security authentication method may also be fingerprint authentication by the user inputting a fingerprint, voice authentication by the user inputting a voice, etc., and is not limited thereto. The first security authentication module receives the password collected by the PC and compares it with the unlock password set by the user on the mobile phone. If the password collected by the PC matches the unlock password set on the mobile phone, the authentication is successful; otherwise, the authentication fails.

[0353] S507: The first security authentication module successfully authenticates the password.

[0354] If password authentication succeeds on the phone, a successful result is returned to the PC. If authentication fails on the phone, a failed result is returned to the PC. Optionally, upon receiving the failed result, the PC can prompt the user in the security authentication window that the password is incorrect. The user can re-enter the password and then initiate a security authentication request to the phone for password verification. Of course, you can set the authentication to suspend if the number of failed authentications exceeds a threshold for incorrect passwords. This threshold can be set by the user or left as the default value.

[0355] S508: The first security authentication module sends a message of successful authentication to the first communication module.

[0356] S509: The first communication module sends an authentication success message to the second communication module based on the second communication connection.

[0357] S510: The second communication module sends an authentication success message to the second security authentication module.

[0358] S511: The second security authentication module sends a message indicating successful authentication to the message center module.

[0359] S512: The message center module controls the display screen to display a transition loading window.

[0360] For example, please refer back to the transition loading window shown in Figure 7, that is, the transition loading interface 706 is displayed in the screen projection window 703. It is understandable that if the authentication is successful, the application screen projection steps of Figures 12 to 15 will be started. It takes a certain amount of time to successfully start the application screen projection, so there is a certain time span between displaying the security authentication window and displaying the application screen projection window. Optionally, in order to improve the user's perception and experience, after the security authentication is successful, the transition loading window can be started and displayed on the target device, which can prompt the user that the security authentication has passed and the application screen projection is being started. After the application screen projection steps of Figures 12 to 15 are successfully started, the transition loading interface displayed in the screen projection window 703 switches to the application interface.

[0361] In other embodiments, after the target device receives a user instruction to project an application screen on the corresponding source device, it first determines whether security authentication is required. For example, whether this is the first time the source device is projecting its screen on the target device. Another example is whether an unlock password has been set on the source device.

[0362] The following describes a screen projection method provided by an embodiment of the present application. This method adds a judgment mechanism when the target device receives a user operation on the message notification control. Please refer to Figure 18, which is an example flow chart for adding security authentication judgment. This method may include but is not limited to the following steps:

[0363] S1: The source device and the target device establish a first communication connection;

[0364] S2: The source device receives the application message;

[0365] S3: The application message flow from the source device is transferred to the target device;

[0366] S4: Displaying a message notification control associated with the application message on the target device;

[0367] S5: When the source device is in a locked or off-screen state, the target device receives a user operation on a message notification control;

[0368] S6: The source device establishes a second communication connection with the target device;

[0369] S7: The target device sends a security authentication request to the source device;

[0370] S8: The source device sends a message of successful authentication to the target device;

[0371] S9: The target device establishes a third communication connection with the source device;

[0372] S10: The target device sends a request to the source device to start application projection;

[0373] S11: The source device starts the application screen projection;

[0374] S12: The source device sends the display data of the application screen projection to the target device;

[0375] S13: The target device displays the application screen projection window.

[0376] As an implementation method, a judgment mechanism for whether security authentication is required is added between step S5 and step S6. In the scenario where the source device projects the screen to the target device for the first time, if the source device has not set an unlock password, the target device does not need to display the unlock verification interface in the projection window, but instead instructs the source device to directly perform heterogeneous projection. In this scenario, the target device can directly display the "prompt waiting message". After the source device completes drawing the application interface to be projected, the target device controls the above-mentioned projection window to display the application interface to be projected.

[0377] When the source device is projecting its screen to the target device for the first time, if the source device has an unlock password set, that is, the user is required to perform password authentication before directly using the source device. In this case, after the target device determines that the source device is qualified for projection, it can trigger security authentication. For a description of security authentication, please refer to the above and will not be repeated here.

[0378] In addition, it should be noted that when the source device performs heterogeneous screen projection to the target device for the first time and an unlock password is set, then regardless of whether the source device is in the locked screen state or unlocked state, the security authentication mechanism will be triggered when the target device instructs the source device to perform heterogeneous screen projection.

[0379] It should be noted that the first time the source device starts screen casting on the target device may refer to the first time the source device starts heterogeneous screen casting on the same target device, or it may refer to the first time the heterogeneous screen casting is started during the period from when the target device is turned on to when it is turned off. Specifically for the second explanation, assuming that the target device is turned on for the first time, and the target device is turned off for the first time after being used for a period of time, the first period from the first power-on to the first power-off is the first period of use. The target device is turned on for the second time, and the target device is turned off for the second time after being used for a period of time, and the second period from the second power-on to the second power-off is the second period of use. During the first period of use, the first time the source device starts heterogeneous screen casting on the target device belongs to the "first" start-up referred to in the second explanation; during the second period of use, the first time the source device starts heterogeneous screen casting on the target device also belongs to the "first" start-up referred to in the second explanation.

[0380] As an implementation method, in a scenario where this is not the first time that the source device is projecting screen to the target device, that is, the source device has previously initiated heterogeneous screen projection on the same target device, or the source device has initiated heterogeneous screen projection on the target device during the period from when the target device was powered on to when it was powered off.

[0381] Specifically, the second explanation is that during the first use period, the source device starts heterogeneous screen projection on the target device for the first time, and when the source device starts heterogeneous screen projection on the target device for the second time, the target device will determine that the source device has started heterogeneous screen projection on the target device; the same applies to the third, fourth, and other times thereafter. During the second use period, the source device starts heterogeneous screen projection on the target device for the first time, and as can be seen from the above, the target device will determine that the source device has started heterogeneous screen projection on the target device for the first time; when the source device starts heterogeneous screen projection on the target device for the second time, the target device will determine that the source device has started heterogeneous screen projection on the target device; the same applies to the third, fourth, and other times thereafter. If the source device has started heterogeneous screen projection on the target device, in order to reduce the number of times the user enters the password while ensuring the privacy of the user, the target device can optionally first determine whether the time since the last security authentication exceeds the preset time threshold.

[0382] For example, when the time since the last security authentication does not exceed the preset time threshold, there is no need to re-authenticate. It should be noted that the preset time threshold can be a default value or a value set by the user, which indicates that within this time range, the screen projection connection between the source device and the target device still meets the requirements of protecting user privacy and security. Therefore, when the target device determines that the time since the last security authentication does not exceed the preset time threshold, the screen projection is started directly without displaying the unlock verification interface. This can avoid repeatedly requiring the user to enter the password, which causes cumbersome operations, and improves the convenience of users using heterogeneous screen projection while protecting user privacy and security.

[0383] As another example, when the time since the last security authentication is passed exceeds the preset time threshold, security authentication needs to be re-performed. It can be understood that when the target device determines that the time since the last security authentication is passed exceeds the preset time threshold, it is considered that the screen projection connection between the source device and the target device at this time no longer meets the requirements for protecting user privacy and security. The security authentication process in this case is the same as the security authentication process of "the source device starts heterogeneous screen projection on the target device for the first time" mentioned above, and will not be repeated here.

[0384] For example, assuming that the source device is in a locked screen state and an unlock password is set, and heterogeneous screen projection has not been started on the target device before. When the target device determines that the user instructs the source device to project the screen, and determines that the source device has the conditions for projecting the screen, it can trigger the security authentication mechanism. After passing the security authentication, the first heterogeneous screen projection is successfully started; the first heterogeneous screen projection ends at time T1; assuming that the preset time threshold set by the user is 1 hour, at time T2, the target device again detects that the user instructs the same source device to perform heterogeneous screen projection; the target device determines that the source device has started heterogeneous screen projection on the target device, and then determines whether the time from T2 to T1 is greater than 1 hour; the target device determines that the value of (T2-T1) is less than 1 hour, and directly starts the second heterogeneous screen projection.

[0385] In other embodiments, after the target device receives the user's instruction to perform heterogeneous screen projection on the corresponding source device (i.e., after step S5), screen projection failure may also occur. For example, when the application corresponding to the message notification control is protected by the application lock, the screen projection fails. For another example, when the channel used for screen projection is occupied by other devices, the screen projection fails. For another example, when the network is unstable, the screen projection fails, etc.

[0386] First of all, application lock refers to the protection set by the user of the source device for a certain application in order to protect the security or privacy of the application, and a pre-set password or fingerprint is required to open the application. Therefore, in the scenario where the source device is in the locked screen state, out of consideration for the security and privacy of the source device, in some possible embodiments, before the actual screen projection, it will be determined whether application a (for example, the application corresponding to the message card selected by the user on the PC) is optionally protected by the source device. If the source device sets an application lock for the application a, it means that the owner of the source device hopes that the application a needs to be unlocked by entering a password or fingerprint every time it is accessed. Then, for such applications with strong privacy protection, it is not appropriate to start the screen projection directly on the target device side when the source device is in the locked screen state, which goes against the original intention of the user to set the application lock.

[0387] Therefore, the embodiment of the present application provides three possible scenarios in which the screen projection startup fails. After the PC detects that the user has selected the message card 702, it sends a query message 1 to the corresponding device (i.e., the mobile phone), and the query message 1 is used to query whether the mobile phone is configured with an application lock for the information application. If the mobile phone is configured with an application lock for the information application, the mobile phone will send feedback information 1 to the PC, and the feedback information 1 indicates that the information application is currently protected by an application lock. In this way, after receiving the feedback information 1, the PC can display a reminder message 1. The reminder message 1 is used to remind the user that the application corresponding to the selected message card is protected by an application lock. For example, it can be the text "The information application is protected by an application lock, please go to the mobile phone to unlock the application lock." In addition, the reminder information can be displayed on the PC in the form of a floating window or a card. The embodiment of the present application does not limit the display form of the reminder information.

[0388] Secondly, the source device and the target device can establish a screen projection connection through WLAN or WI-FI P2P. The establishment of the screen projection connection can refer to the method of establishing a communication connection between the source device and the target device above, which will not be repeated here. For the convenience of description, the following embodiment establishes a screen projection connection in WI-FI P2P mode.

[0389] In some embodiments of the present application, after the PC detects that the user has selected the message card 702, it determines whether the screen projection communication channel (Wi-Fi P2P channel) between the source device (i.e., mobile phone) and the target device (i.e., PC) corresponding to the message card 702 has been occupied by other services. Among them, the source device and the target device establish a screen projection connection through the Wi-Fi P2P channel to transmit data for the screen projection service, thereby realizing the mirroring of content between two or more electronic devices. When the source device and the target device have established other services based on WI-FI P2P, such as PC collaboration, device sharing, etc., then the channel for the screen projection service between the source device and the opposite device (i.e., the screen projection channel) is occupied, resulting in the source device being unable to project the screen to the target device based on WI-FI P2P.

[0390] For example, if the PC determines that the screen projection channel (Wi-Fi P2P) between the PC and the mobile phone is occupied, the screen projection fails to start, and a reminder message 2 is displayed. This reminder message 2 exemplarily prompts the user to "screen projection connection conflict, please disconnect the connection with other devices and try again", prompting the user to go to the source device (mobile phone) or target device (PC) to disconnect the screen projection connection with other devices. The reminder message can be displayed on the PC in the form of a floating window or card. The embodiment of the present application does not limit the display format of the reminder message.

[0391] Again, in some embodiments of the present application, after the PC detects that the user has selected the message card 702, it determines whether the network status of the source device (i.e., the mobile phone) corresponding to the message card meets the screen projection requirements. Among them, the network status includes the flow table matching status, the data packet execution port load status, the total number of lost / forwarded / received data packets or the failure of the data packet TTR and other similar situations. As long as any network status that may occur in the network is included, it is not limited here. It can be understood that the network status will affect the service quality of the screen projection, and the PC can optionally have a minimum network status threshold that meets the screen projection conditions. If the PC determines that the current network status is lower than this minimum network status threshold, it is considered that the current network status cannot meet the screen projection requirements, resulting in a screen projection failure.

[0392] If the PC determines that the current network status does not meet the screen projection requirements, the screen projection fails and prompt message 3 is displayed. This prompt message 3 can remind the user to wait until the network is stable before trying to trigger the source device to cast the screen. For example, the prompt "Network is unstable, please try again" prompts the user to adjust and optimize the network status.

[0393] It is understandable that the above three scenarios are only examples and are not intended to limit the embodiments of this application. The PC's determination of whether the corresponding application is protected by an application lock, whether the screen projection channel is occupied, and whether the network status meets the screen projection requirements is not a required operation of the embodiments of this application, and the order of determination is not limited.

[0394] In some embodiments, please refer back to Figure 18. After step S5 of Figure 18, it is possible to add a judgment on whether there are circumstances that may cause the screen projection to fail, which may include but is not limited to the above three circumstances. In response to the user clicking the message card 702, the PC judges in each judgment node in turn whether the application lock is set, whether the screen projection channel is occupied, and whether the network status meets the screen projection requirements. For example, as long as there is a judgment result that triggers the above-mentioned scenario of failure to start the screen projection, this round of judgment is terminated. After the user adjusts according to the prompts, he can click the message notification control again to trigger a new round of judgment. Of course, in some embodiments, the judgment order of the judgment mechanism can be adjusted, and after a judgment node triggers the failure to start the screen projection, the judgment process can stay at the judgment node and continue to detect whether the conditions for entering the next judgment node are met. The user adjusts according to the prompts, and when the conditions for entering the next judgment node are met, the judgment process automatically enters the next judgment node.

[0395] It is understandable that the PC can first check whether the corresponding application is protected by the application lock (which can be called application lock detection), then check whether the communication channel used for screen projection is occupied (which can be called channel detection), and finally check whether the network conditions meet the requirements (referred to as network detection). In fact, in the embodiments of the present application, the order of the above-mentioned multiple tests is not limited. That is, in some embodiments, the PC can also first check whether the communication channel used for screen projection is occupied, then check whether the corresponding application is protected by the application lock, and finally check the network conditions.

[0396] In other possible embodiments, the PC may also simultaneously perform application lock detection, channel detection, and network detection, and display a corresponding reminder message when the test results indicate that at least one of the test items has failed to trigger screen projection. For example, if the test results indicate that the application is protected by application lock and the communication channel is occupied, the PC will simultaneously display reminder message 1 and reminder message 2.

[0397] In addition, when the message card 702 is displayed on the PC desktop 701, if the PC determines that the user has selected the message card 702, then the PC also needs to perform an application lock test, a channel test, and a network test on the mobile phone. If the test results indicate that none of the above test items have triggered a screen projection failure, the PC determines that the mobile phone has the conditions for screen projection, so the PC can display the application interface 704 in the screen projection window 703. In the event of screen projection failure, the PC can display a corresponding reminder message.

[0398] The above describes in detail the screen projection method and examples of electronic devices provided in the embodiments of the present application. It is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0399] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each function can be divided into various functional modules, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0400] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0401] The electronic device provided in this embodiment is used to execute the above-mentioned screen projection method, and thus can achieve the same effect as the above-mentioned implementation method.

[0402] When integrated, the electronic device may also include a processing module, a storage module, and a communication module. The processing module may be used to control and manage the operation of the electronic device. The storage module may be used to support the execution of program code and data stored in the electronic device. The communication module may be used to support communication between the electronic device and other devices.

[0403] The processing module may be a processor or a controller. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module may be a memory. The communication module may specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, or a Wi-Fi chip.

[0404] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a device having the structure shown in FIG. 9 .

[0405] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the screen projection method of any of the above embodiments.

[0406] An embodiment of the present application also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement the screen projection method in the above-mentioned embodiment.

[0407] In addition, embodiments of the present application further provide a device, which may be a chip, component, or module. The device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions. When the device is running, the processor may execute the computer-executable instructions stored in the memory to cause the chip to perform the caller ID method in each of the above-mentioned method embodiments. The electronic device, computer-readable storage medium, computer program product, or chip provided in this embodiment is used to perform the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding method provided above and will not be repeated here.

[0408] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0409] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0410] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0411] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0412] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. The above embodiments are only used to illustrate the technical solution of the present application, not to limit it; although the present application is described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that it can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A screen projection system, characterized in that: The system includes a first electronic device and a second electronic device, wherein the first electronic device includes a first application. The first electronic device is used to receive first data of the first application; The first electronic device is used to send a first message including the first data to the second electronic device; the second electronic device, configured to display a first control in response to receiving the first message; The second electronic device is used to receive a first operation of the user on the first control at a first time; The second electronic device is configured to send a screen projection request to the first electronic device in response to the first operation; The first electronic device is configured to send image information of the first application to the second electronic device in response to receiving the screen projection request, where the image information of the first application includes the first data; the second electronic device being configured to display a window according to the received image information of the first application, wherein a first interface of the first application is displayed in the window at a second time, the first interface including the image information of the first application; The second time is later than the first time, and during the time interval between the first time and the second time, the first electronic device is in a screen-locked or screen-off state.

2. The system according to claim 1, wherein: During the first time interval, the first electronic device is further configured to display a main interface; The first electronic device is further configured to receive a second operation from the user; In response to receiving the second operation, the first electronic device is in a screen-locked or screen-off state during a second time interval, where the start time of the second time interval is the end time of the first time interval; The first electronic device is further configured to receive a third operation from the user; In response to receiving the third operation, the first electronic device displays the main interface in a third time interval, where the start time of the third time interval is the end time of the second time interval; During the third time interval, the second electronic device is further configured to display the first interface in the window.

3. The system according to claim 1 or 2, characterized in that The second electronic device does not include the first application.

4. The system according to any one of claims 1 to 3, characterized in that: The first electronic device further includes a second application, and the first application and the second application are different; The first electronic device is further configured to receive an operation from a user to open the second application; In response to the user opening the second application, the first electronic device is further configured to display a second interface of the second application in a fourth time interval, where the start time of the fourth time interval is the end time of the third time interval; During the fourth time interval, the second electronic device is further configured to display the first interface in the window.

5. The system according to any one of claims 1 to 4, characterized in that: In response to receiving a user operation on the first interface, the second electronic device is further configured to display a third interface of the first application in the window, where the third interface is different from the first interface.

6. The system according to any one of claims 1 to 3, characterized in that: The first electronic device further includes a third application, which is different from the first application; The first electronic device is further configured to receive second data of the third application; The first electronic device is further configured to send a second message including the second data to the second electronic device; In response to receiving the second message, the second electronic device is further configured to display a second control; The second electronic device is further configured to receive a fourth operation on the second control by the user at a third time; In response to the fourth operation, the second electronic device is further configured to send a screen projection request to the first electronic device; In response to receiving the screen projection request, the first electronic device sends image information of the third application to the second electronic device, where the image information of the third application includes the second data; In response to receiving the image information of the third application, at a fourth time, the second electronic device displays a fourth interface of the third application in the window, where the fourth interface includes the image information of the third application; The fourth time is later than the third time, and during the time interval between the third time and the fourth time, the first electronic device is in a screen-locked or screen-off state.

7. The system according to any one of claims 1 to 6, characterized in that: In response to receiving the user's operation on the first message on the first interface, the second electronic device is further used to send an instruction to the first electronic device to instruct the first electronic device to perform the operation.

8. The system according to any one of claims 1 to 7, characterized in that: The second electronic device is further configured to display the first control on the desktop; or, the second electronic device is further configured to display the first control in a message list window, where the first control is a message card corresponding to the first message.

9. The system according to any one of claims 1 to 8, characterized in that: The first control includes the content of the first message.

10. The system according to any one of claims 1 to 9, characterized in that: The first operation is an operation of a user clicking the first control; or the first operation is an operation of a user dragging the first control.

11. The system according to any one of claims 1 to 10, characterized in that: In response to receiving the first operation, the second electronic device is further configured to display a security authentication interface in the window; Upon receiving an operation in which a user inputs authentication information on the security authentication interface, the second electronic device is further configured to send the authentication information to the first electronic device; Upon receiving the authentication information, the first electronic device is further configured to determine that the authentication information meets a preset value, and send a message of successful authentication to the second electronic device; In response to receiving the message indicating successful authentication, the second electronic device sends a screen projection request to the first electronic device.

12. The system according to any one of claims 1 to 11, characterized in that: The first electronic device is configured to send the first message to the second electronic device via a first communication connection; The second electronic device is used to send the screen projection request to the first electronic device through a second communication connection; The first electronic device is configured to send the image information of the first application to the second electronic device through the second communication connection; The first communication connection is different from the second communication connection.

13. The system according to claim 11, wherein: The second device is further configured to send the authentication information to the first electronic device via a third communication connection; The first electronic device is further configured to send the authentication information to the second electronic device via the third communication connection. The message of successful certification; The third communication connection is different from the first communication connection and different from the second communication connection.

14. A screen projection method, characterized in that: The method is applied to a first electronic device, where the first electronic device includes a first application: The first electronic device receives first data of the first application; The first electronic device sends a first message including the first data to the second electronic device through a first communication connection; In response to receiving a screen projection request sent by the second electronic device through the second communication connection at a first time, the first electronic device sends image information of the first application to the second electronic device through the second communication connection, so that the second electronic device displays the first interface in the window at a second time; The image information of the first application includes the first data, and the first interface includes the image information of the first application; The second time is later than the first time, and during the time interval between the first time and the second time, the first electronic device is in a screen-locked or screen-off state.

15. The method according to claim 14, characterized in that After the first electronic device receives the screen projection request, the method includes: The first electronic device sets one or more activities of the first application to an OnResume state; The first electronic device controls one or more activities of the first application not to enter a sleep process.

16. The method according to claim 15, characterized in that The first electronic device includes an asynchronous processor, an activity recorder, and a root window container. In the first time interval, after the first electronic device receives the screen projection request, the method further includes: The asynchronous processor marks one or more activities of the first application with a first tag; The asynchronous processor sends an instruction to start an activity with the first tag; Upon receiving the instruction to start the activity with the first tag, the activity recorder determines that one or more activities of the first application have the first tag, and sets the one or more activities of the first application to an OnResume state; The root window container determines that one or more activities of the first application have the first label, and controls the one or more activities of the first application not to enter a dormant process.

17. The method according to any one of claims 14 to 16, characterized in that: The first electronic device includes a virtual display area and a visible display area. Before the first electronic device sends the first interface to the second electronic device, the method further includes: The first electronic device marks the virtual display area with a second label; The first electronic device draws a first image in the virtual display area, where the first image is an interface of the first application corresponding to the first message; The first electronic device draws a second image in the visible display area, where the second image is an interface corresponding to when the first electronic device is in the locked screen or off screen state; The first electronic device determines that the virtual display area has the second tag, acquires the first image to obtain a first interface, and the first interface does not include the second image.

18. The method according to any one of claims 14 to 17, characterized in that: The method further comprises: During a first time interval, the first electronic device displays a main interface; The first electronic device receives a second operation from the user; In response to receiving the second operation, the first electronic device is in a screen-locked or screen-off state during a second time interval, where the start time of the second time interval is the end time of the first time interval; The first electronic device receives a third operation from the user; In response to receiving the third operation, the first electronic device draws a main interface in the visible display area and displays the main interface in a third time interval, where the start time of the third time interval is the end time of the first time interval; In the third time interval, the first electronic device determines that the virtual display area has the second tag, does not obtain the main interface of the visible display area and sends it to the second electronic device, so that the second electronic device displays the first interface in the window.

19. The method according to any one of claims 14 to 18, characterized in that: Before the first electronic device receives the request to start screen projection, the method further includes: The first electronic device receives authentication information sent by the second electronic device through the third communication connection; The first electronic device determines that the authentication information meets a preset value, and sends a message of successful authentication to the second electronic device through a third communication connection, so that the second electronic device sends a screen projection request to the first electronic device.

20. The method according to any one of claims 14 to 19, characterized in that: The method comprises: The first electronic device receives the instruction sent by the second electronic device and executes the user's operation on the first message on the first interface.

21. A screen projection method, characterized in that: The method is applied to a second electronic device, and includes: The second electronic device receives a first message sent by the first electronic device through the first communication connection, where the first message includes first data of a first application of the first electronic device; In response to receiving the first message, the second electronic device displays a first control; The second electronic device receives a first operation of the user on the first control at a first time; In response to the first operation, the second electronic device sends a screen projection request to the first electronic device through a second communication connection; The second electronic device receives image information of the first application sent by the first electronic device through the second communication connection, where the image information of the first application includes the first data; The second electronic device displays a window according to the image information of the first application, wherein a first interface of the first application is displayed in the window at a second time, and the first interface includes the image information of the first application; The second time is later than the first time, and during the time interval between the first time and the second time, the first electronic device is in a screen-locked or screen-off state.

22. The method according to claim 21, characterized in that The method further comprises: During a first time interval, the second electronic device displays the first interface in the window, the first time interval being a time interval during which the first electronic device displays the main interface, and a start time of the first time interval being later than the second time; During the second time interval, the second electronic device displays the first interface in the window. The second time interval is the time interval during which the first electronic device displays the second interface of the second application. The start time of the first time interval is the end time of the first time interval.

23. The method according to claim 21 or 22, characterized in that The method comprises: The second electronic device does not include the first application.

24. The method according to any one of claims 21 to 23, characterized in that The method comprises: In response to receiving a user operation on the first interface, the second electronic device is further configured to display a third interface of the first application in the window, where the third interface is different from the first interface.

25. The method according to any one of claims 21 to 24, characterized in that The method further comprises: The second electronic device receives a second message sent by the first electronic device through the third communication connection, where the second message includes second data of a third application of the first electronic device; In response to receiving the second message, the second electronic device displays a second control; The second electronic device receives a second operation on the second control by the user at a third time; In response to the second operation, the second electronic device sends a screen projection request to the first electronic device through a fourth communication connection; The second electronic device receives the image information of the third application sent by the first electronic device through the fourth communication connection, where the image information of the third application includes the second data; The second electronic device displays a window according to the image information of the third application, and a fourth interface of the third application is displayed in the window at a fourth time, the fourth interface including the image information of the third application; The fourth time is later than the third time, and during the time interval between the third time and the fourth time, the first electronic device is in a screen-locked or screen-off state.

26. The method according to any one of claims 21 to 25, characterized in that The method comprises: In response to receiving the user's operation on the first message on the first interface, the second electronic device sends an instruction to the first electronic device to instruct the first electronic device to perform the operation.

27. The method according to any one of claims 21 to 26, characterized in that The method comprises: The second electronic device displays the first control on the desktop; or, the second electronic device displays the first control in a message list window, where the first control is a message card corresponding to the first message.

28. The method according to any one of claims 21 to 27, characterized in that The method also includes: the first control including content of the first message.

29. The method according to any one of claims 21 to 28, characterized in that The method further includes: the first operation is an operation of a user clicking the first control; or the first operation is an operation of a user dragging the first control.

30. The method according to any one of claims 21 to 29, characterized in that The method further includes: in response to receiving the first operation, the second electronic device displaying a security authentication interface in the window; Upon receiving an operation in which the user inputs authentication information on the security authentication interface, the second electronic device sends the authentication information to the first electronic device via a third communication connection; The second electronic device receives a message indicating successful authentication sent by the first electronic device through the third communication connection; in response to receiving the message indicating successful authentication; The second electronic device receives the message of successful authentication and sends a screen projection request to the first electronic device.

31. The method according to claim 30, wherein After the second electronic device receives the first operation, the method further includes: The second electronic device determines that the first application is projected on the second electronic device for the first time, and the first electronic device is set with security authentication, and displays the security authentication interface in the projection window.

32. The method according to claim 30, wherein After the second electronic device receives the first operation, the method further includes: The second electronic device determines that this is not the first time that the first application is projected on the second electronic device, and the current time is longer than a preset time from the last successful authentication. The first electronic device is set with security authentication, and the security authentication interface of the first electronic device is displayed in the projection window.

33. The method according to any one of claims 21 to 32, characterized in that After the second electronic device receives the first operation, the method further includes: The second electronic device determines that the first application is set with an application lock, and the second electronic device does not display the screen projection window of the first electronic device.

34. An electronic device, characterized in that: include: one or more processors; one or more memories; The one or more memories store one or more computer programs, and the one or more computer programs include instructions, which, when executed by the one or more processors, enable the electronic device to perform the method according to any one of claims 14 to 20.

35. An electronic device, characterized in that: include: one or more processors; one or more memories; The one or more memories store one or more computer programs, and the one or more computer programs include instructions, which, when executed by the one or more processors, enable the electronic device to perform the method according to any one of claims 21 to 33.

36. A computer-readable storage medium, characterized in that The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 14-20 or 21-33.

Images