Screen projection display method and system and electronic equipment

Abstract

The invention provides a projection screen display method and system and electronic device.The method is applied to a first electronic device, the first electronic device and a second electronic device establish projection screen connection, the method comprises the steps that a security layer in a to-be-synthesized layer of a to-be-projected screen interface is determined, and the mark of the security layer is a first numerical value; changing the first numerical value into a second numerical value, and synthesizing the to-be-synthesized image layer according to the second numerical value, so that a display area corresponding to the synthesized safety image layer is normally displayed; and transmitting screen projection data to the second electronic equipment, wherein the screen projection data comprises the synthesized image layer. According to the method and the device, by changing the numerical value in the flag bit of the security layer, only the synthesis mode of the security layer can be changed without changing other attributes of the security layer, so that the security of the security layer can be ensured under the condition that the display area corresponding to the security layer can be normally displayed.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and more specifically, to a method, system, and electronic device for screen mirroring display. Background Art

[0002] With the development of application display technology, more and more electronic devices support screen mirroring technology. Among them, screen mirroring technology is to mirror the application interface launched on an electronic device with a smaller display screen (such as the first electronic device) to another electronic device with a larger display screen (such as the second electronic device), so as to achieve a better viewing effect and to be able to operate the application interface using the operating device on the second electronic device.

[0003] However, when the application interface launched on the first electronic device includes a security layer, when the application interface is mirrored to the second electronic device, the display area corresponding to the security layer is black, resulting in the user being unable to view the information corresponding to the security layer and even more unable to operate the display interface corresponding to the security layer, greatly reducing the user experience. Summary of the Invention

[0004] This application provides a method, system, and electronic device for screen mirroring display. By changing the value of the flag bit of the security layer in the layer to be synthesized, the security layer can be synthesized normally, thus solving the problem in the related art that the user cannot recognize the display area corresponding to the security layer and improving the user experience.

[0005] In a first aspect, a method for screen mirroring display is provided. This method is applied to the first electronic device. The first electronic device and the second electronic device establish a screen mirroring connection. The execution subject of this method can be the first electronic device or a chip installed in the first electronic device. This method includes: determining the security layer in the layer to be synthesized of the interface to be mirrored, where the flag bit of the security layer is a first value; changing the first value to a second value; synthesizing the layer to be synthesized according to the second value so that the display area corresponding to the security layer is normally displayed; and sending the screen mirroring data including the synthesized layer to the second electronic device.

[0006] The method for screen mirroring display provided by the first aspect is that the first electronic device searches for the secure layer in the layer to be synthesized corresponding to the interface to be displayed, and then changes the first value of the flag bit of the secure layer to the second value, so that when the first electronic device synthesizes the layer to be synthesized, the secure layer is synthesized normally. The display area corresponding to the secure layer synthesized in this way can be displayed normally. By changing the value of the flag bit, this method can only change the synthesis method of the secure layer without changing other attributes of the secure layer. For example, the display area corresponding to the secure layer cannot be captured, cannot be recorded, and cannot be multitasked. Thus, while ensuring the normal display of the secure layer, the security of the secure layer is guaranteed.

[0007] In a possible implementation manner of the first aspect, before determining the secure layer in the layer to be synthesized of the interface to be screen mirrored, the method further includes: determining that the screen mirroring between the first electronic device and the second electronic device is non-secure. In the related art, when the screen mirroring between the first electronic device and the second electronic device is non-secure, the display area corresponding to the secure layer is black. Therefore, when it is determined that the screen mirroring between the first electronic device and the second electronic device is non-secure, the screen mirroring display method provided by this application can be used, so as to view the display area corresponding to the secure layer while ensuring the security of the layer.

[0008] For example, when the first electronic device and the second electronic device are wirelessly connected, the first electronic device is in a non-secure screen mirroring state.

[0009] For another example, when the flag bit in the interface to be screen mirrored is used to indicate that the interface to be screen mirrored is a non-secure interface, the first electronic device is in a non-secure screen mirroring state.

[0010] In a possible implementation manner of the first aspect, in the synthesized layer, the display area corresponding to the secure layer cannot be captured, cannot be recorded, and cannot be multitasked. This implementation manner can ensure the security of the secure layer while ensuring the normal display of the secure layer.

[0011] In a possible implementation manner of the first aspect, the window management service WMS in the first electronic device can change the first value to the second value.

[0012] In a possible implementation manner of the first aspect, the interface synthesis system surfaceflinger in the first electronic device can synthesize the layer to be synthesized according to the second value.

[0013] In a possible implementation manner of the first aspect, the above-mentioned first value can be 0x80, and the second value can be 0x30.

[0014] In a possible implementation of the first aspect, the first electronic device projects the screen to the second electronic device in a manner of homologous screen projection or heterologous screen projection.

[0015] In a second aspect, a screen projection display system is provided. The system includes a first electronic device and a second electronic device. The first electronic device is used to project the screen to the second electronic device. The first electronic device is used to determine a secure layer in the layers to be synthesized of the interface to be projected. The flag bit of the secure layer is a first value. The first electronic device is further used to change the first value to a second value. The first electronic device is further used to synthesize the layers to be synthesized according to the second value, so that the display area corresponding to the synthesized secure layer is normally displayed. The first electronic device is further used to synthesize the layers to be synthesized according to the second value, so that the display area corresponding to the secure layer is normally displayed. The first electronic device is further used to send screen projection data to the second electronic device, and the screen projection data includes the synthesized layers.

[0016] In the screen projection display system provided in the second aspect, by searching for the secure layer in all the layers to be synthesized of the interface to be displayed in the first electronic device, and then changing the first value of the flag bit of the secure layer to the second value, when the first electronic device synthesizes all the layers to be synthesized, the secure layer is normally synthesized. Finally, after the screen is projected to the second electronic device, the second electronic device can normally display the area corresponding to the secure layer. By changing the value of the flag bit in this way, only the synthesis method of the secure layer can be changed without changing other attributes of the secure layer. For example, the display area corresponding to the secure layer cannot be captured, cannot be recorded, and cannot be multitasked. Thus, while ensuring the normal display of the secure layer, the security of the secure layer is ensured.

[0017] In a possible implementation of the second aspect, before the first electronic device determines the secure layer in the layers to be synthesized of the interface to be projected, the first electronic device is further used to determine that the screen projection between the first electronic device and the second electronic device is an insecure screen projection. In the related art, when the screen projection between the first electronic device and the second electronic device is an insecure screen projection, the display area corresponding to the secure layer is black. Therefore, when it is determined that the screen projection between the first electronic device and the second electronic device is an insecure screen projection, the screen projection display method provided in this application can be used to view the display area corresponding to the secure layer while ensuring the security of the layer.

[0018] For example, when the first electronic device and the second electronic device are wirelessly connected, the first electronic device is in an insecure screen projection state.

[0019] For another example, when the flag bit in the interface to be projected is used to indicate that the interface to be projected is an insecure interface, the first electronic device is in an insecure screen projection state.

[0020] In a possible implementation of the second aspect, in the synthesized layer, the display area corresponding to the secure layer cannot be captured by screenshots, cannot be recorded by screen recording, and cannot be multitasked for display. This implementation can ensure the security of the secure layer while ensuring its normal display.

[0021] In a possible implementation of the second aspect, the window management service (WMS) in the first electronic device can change the first value to the second value.

[0022] In a possible implementation of the second aspect, the interface composition system (surfaceflinger) in the first electronic device can compose the layers to be composed according to the second value.

[0023] In a possible implementation of the second aspect, the above first value can be 0x80, and the above second value can be 0x30.

[0024] In a possible implementation of the second aspect, the first electronic device projects the screen to the second electronic device in a way of homologous screen projection or heterologous screen projection.

[0025] In a possible implementation of the second aspect, the first electronic device is used to encode the synthesized layer to obtain the encoded layer; the first electronic device is also used to package the encoded layer according to the Real-time Transport Protocol (RTP) or the RTP Control Protocol (RTCP); the first electronic device is used to send the packaged layer to the second electronic device. In this implementation, the first electronic device will encode and compress all the layers of the layer to be composed, and perform RTP or RTCP packaging on the encoded layer, so as to send the packaged layer to the second electronic device. This can not only improve the security of the layer data, but also reduce the data transmission volume and improve the data transmission efficiency.

[0026] In the third aspect, a communication device is provided, and the communication device includes units for performing each step in the above first aspect or any possible implementation of the first aspect.

[0027] In the fourth aspect, a communication device is provided, and the communication device includes at least one processor and a memory. The processor and the memory are coupled, and the memory stores program instructions. When the program instructions stored in the memory are executed by the processor, the methods in the above first aspect or any possible implementation of the first aspect are executed.

[0028] In the fifth aspect, a communication device is provided, and the communication device includes at least one processor and an interface circuit. The at least one processor is used to execute the methods in the above first aspect or any possible implementation of the first aspect.

[0029] In a sixth aspect, an electronic device is provided, which includes the communication device provided in the second aspect above, or the electronic device includes the communication device provided in the third aspect above, or the electronic device includes the communication device provided in the fourth aspect above.

[0030] In a seventh aspect, a computer program product is provided, which includes a computer program that, when executed by a processor, is used to execute the method in the first aspect above or any possible implementation manner of the first aspect.

[0031] In an eighth aspect, a computer-readable storage medium is provided, in which a computer program is stored, and when the computer program is executed, it is used to execute the method in the first aspect above or any possible implementation manner of the first aspect.

[0032] In a ninth aspect, a chip is provided, which includes: a processor for calling and running a computer program from a memory, so that a communication device equipped with the chip executes the method in the first aspect above or any possible implementation manner of the first aspect. Description of the Drawings

[0033] Figure 1 A schematic flowchart showing the display of interface information by an electronic device in a screen mirroring scenario in the related art is shown;

[0034] Figure 2 A schematic flowchart showing the synthesis process of layers to be synthesized on a first electronic device is shown;

[0035] Figure 3 A schematic diagram of the same-source screen mirroring provided by an embodiment of the present application is shown;

[0036] Figure 4 A schematic diagram of the different-source screen mirroring provided by an embodiment of the present application is shown;

[0037] Figure 5 A schematic diagram of the structure of the electronic device 100 is shown;

[0038] Figure 6 A software structure block diagram of the electronic device provided by an embodiment of the present application is shown;

[0039] Figure 7 A schematic flowchart of the screen mirroring display method provided by an embodiment of the present application is shown;

[0040] Figure 8 A schematic diagram of the display interface of the security layer in the same-source screen mirroring provided by an embodiment of the present application is shown;

[0041] Figure 9Shows a schematic diagram of the display interface of the security layer under heterogeneous screen mirroring provided by an embodiment of the present application;

[0042] Figure 10 Shows a schematic diagram of a chip system provided by an embodiment of the present application. Detailed implementation manners

[0043] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application.

[0044] The terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to limit the present application. As used in the specification and claims of the present application, the singular forms "a", "an", "the", "above", "said", "this" are also intended to include expressions such as "one or more", unless there is a clear indication to the contrary in the context. It should also be understood that in the embodiments of the present application, "one or more" means one or more than two (including two); " / ", describing the association relationship of associated objects, indicates that three relationships can exist; for example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are an "or" relationship.

[0045] Referring to "one embodiment" or "some embodiments" described in this specification means that specific features, structures, or characteristics described in conjunction with the embodiment are included in one or more embodiments of the present application. Thus, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments" and the like that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0046] Multiple related to the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the terms "first", "second", etc. are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

[0047] With the development of Internet technology, screen mirroring technology has been widely applied. Among them, screen mirroring refers to projecting the interface content displayed on a first electronic device into a second electronic device, so that the second electronic device can also synchronously display the interface content displayed on the first electronic device. Through screen mirroring technology, the interface content of a device with a smaller display screen (such as a mobile phone, a tablet computer) can be projected onto a large-screen display device (such as a TV, an in-vehicle multimedia display screen), so as to achieve a better viewing effect and facilitate the user to operate using the operation device of the large-screen device.

[0048] However, when the interface displayed on the first electronic device includes a security layer, when the interface on the first electronic device is projected onto the second electronic device, the display area of the security layer corresponding to the screen mirroring interface displayed on the second electronic device is black (displayed in black). Therefore, the user cannot identify the display content in this display area on the second electronic device.

[0049] It should be understood that an interface is composed of multiple layers.

[0050] Next, taking a screen mirroring initiating device, that is, a source device, and a screen mirrored device, that is, a target device as an example, the process of displaying screen mirroring interface information in the related art will be described. Figure 1 The following shows a schematic flowchart of an electronic device displaying interface information in a screen mirroring scenario in the related art, as Figure 1 shown. Assume that the first electronic device is the source device and the second electronic device is the target device. That is to say, the first electronic device can project the display content in its display interface into the second electronic device for display at the same time. Among them, in addition to completing its own interface display, the first electronic device also needs to synthesize the display content to be displayed on the second electronic device.

[0051] Specifically, when starting screen mirroring, the interface composition system (Surfaceflinger) in the first electronic device will create a virtual screen, and the interface information carried on this virtual screen can be sent to the second electronic device through the network. After Surfaceflinger obtains the layers to be synthesized corresponding to the display interface of the first electronic device, it synthesizes the layers, and then sends the synthesized layers to the hardware graphics renderer (hardwarecomposer, HWC) in the first electronic device. HWC then performs the synthesis, optimization, and rendering of the remaining layers, and sends the layer data after synthesis, optimization, and rendering to the LCD for the first electronic device to display the interface corresponding to the synthesized layer.

[0052] In addition, SurfaceFlinger in the first electronic device also needs to send the layers to be synthesized corresponding to the interface information to be displayed on the virtual screen to the graphics processing unit (GPU) in the first electronic device for the synthesis of the remaining layers.

[0053] After the GPU in the first electronic device completes the synthesis of the layers, it will output the data to be encoded. For example, the format of the data to be encoded can be RGBA8888, and send the data to be encoded to the encoder. After the encoder performs encoding and compression, according to the real-time transport protocol (RTP) or the RTP control protocol (RTCP), the encoded and compressed data is packetized, and then through the transmission control protocol (TCP) or the user datagram protocol (UDP), the packetized data is sent to the second electronic device. For example, the packetized data can be sent to the second electronic device by wireless transmission.

[0054] After the second electronic device receives the packetized data sent by the first electronic device, it will unpack the received data according to RTP or RTCP and send the unpacked data to the decoder for decoding, so as to obtain the decoded content, that is, the interface information. The second electronic device sends the obtained interface information to the display system to display the interface information through the LCD.

[0055] Next, when the layers to be synthesized include a security layer, the specific process of how SurfaceFlinger synthesizes the layers to be synthesized will be described.

[0056] Exemplarily, the graphics rendering framework of the first electronic device can be the graphics pipeline rendering framework of the Android system. Of course, the software system can also include but is not limited to Symbian Apple BlackBerry HarmonyOS and other operating systems, which are not limited in this application. When the graphics rendering framework of the first electronic device is the graphics pipeline rendering framework of the Android system, SurfaceFlinger synthesizes the layers to be synthesized through the Figure 2 synthesis process shown below. The following will be specifically described in conjunction with Figure 2 this.

[0057] It can be understood that each window of an Android application corresponds to a canvas, that is, a Surface. This Surface can be understood as an interface for the producers and users of graphic data to exchange buffer areas. Each Surface contains multiple buffers internally. The transfer of graphics uses the buffer as a carrier, and the Surface is a further encapsulation of the buffer.

[0058] As Figure 2 shown, when the user starts a certain application for screen mirroring, the application in the electronic device will first apply for the handle (Surface) of the native buffer to determine the graphics to be displayed, such as size, position, and display content, etc. The Android system internally generates multiple application graphic data in the interface that needs to be displayed according to this application. One Surface corresponds to one application graphic data, and each application graphic data corresponds to a buffer queue (Buffer Queue).

[0059] It should be understood that each Surface includes application graphic data, and the Surface and the application graphic data form a layer.

[0060] Exemplarily, the display interface includes 3 pieces of application graphic data, or it can be said that the display interface includes 3 Surfaces. Each Surface corresponds to a layer, and each layer corresponds to a Buffer Queue, that is, Surface1 → application graphic data 1 → Buffer Queue1 → layer 1; Surface2 → application graphic data 2 → Buffer Queue2 → layer 2; Surface3 → application graphic data 3 → Buffer Queue3 → layer 3. Among them, → means corresponding to.

[0061] It should be noted that the Buffer Queue is equivalent to an adhesive between Android graphic components. It can be understood as a buffer queue, adopting the producer - consumer mode, and can adjust the fixed period of the buffer from the production queue to the consumption queue. Once the producer hands over its buffer, the Surfaceflinger will be responsible for synthesizing all the content into the display part. Among them, the Buffer Queue can also be understood as a combination of multiple buffers.

[0062] Then, Surfaceflinger synthesizes the generated Buffer Queue by rendering according to the window information provided by the window manager service (WMS), that is, synthesizes all layers (corresponding to Surfaces). It should be understood that the Buffer Queue is the link connecting the Surface and the layer.

[0063] In the related art, when the layer to be synthesized includes a secure layer, Surfaceflinger will empty the data in the Buffer Queue corresponding to the secure layer before synthesis.

[0064] Exemplarily, when the layer 1 corresponding to Surface1 is a secure layer, then when Surfaceflinger synthesizes layer 1, layer 2, and layer 3, it empties the data in Buffer Queue1 in Surface1 and then performs the synthesis of the layers. It is equivalent to the display information in layer 1 being empty, and the data of the remaining layers (layer 2 and layer 3) being retained normally.

[0065] Finally, after the first electronic device completes the synthesis of the layers, it sends them to the second electronic device. After partial layer processing and rendering optimization are performed by the HWC on the second electronic device, the HWC sends the finally rendered and optimized layers to the display system of the second electronic device for display. It can be understood that during the layer synthesis process, the display data corresponding to the secure layer is emptied, and the data corresponding to the secure layer on the display interface is empty. Therefore, on the second electronic device, the display area corresponding to the secure layer seen by the user is a black screen.

[0066] It should be understood that according to different screen mirroring requirements, the screen mirroring methods can include the same-source screen mirroring and different-source screen mirroring. The above synthesis method for the layer to be synthesized including the secure layer can be applied to both the same-source screen mirroring and different-source screen mirroring.

[0067] Among them, the same-source screen mirroring means mirroring one or more application interfaces started on the first electronic device to the second electronic device. In the same-source screen mirroring mode, the first electronic device uses one-way encoding to send a standard video stream encoded from the Surfaces corresponding to all application interfaces rendered on the main screen to the second electronic device, so as to display all application interfaces rendered on the main screen on the display screen of the second electronic device. That is, the same-source screen mirroring is to mirror the default interface of the first electronic device to the display of the second electronic device, such as mirror screen mirroring, etc.

[0068] Figure 3 Shows a schematic diagram of the same-source screen mirroring provided by an embodiment of the present application. Among them, Figure 3(a) in shows the current output interface 301 of the smartphone, and the content 301 of this interface is the home page interface of the smartphone. In response to the user clicking on the Taobao application icon (such as Figure 3 the "Taobao" icon shown in (a) of ), after the operation, as Figure 3 shown in (b) of , the smartphone renders the home page interface 302 and the Taobao login interface 3021 included on the home page interface 302 together on the main screen. Based on the same-source screen mirroring, after the large-screen device receives the standard video stream corresponding to the Surface encoding of the application interface rendered on the main screen of the smartphone, the large-screen device displays the home page interface 303 on the main screen and the Taobao login interface 3031 included on the home page interface 303 according to this standard video stream. As Figure 3 shown in (c) of , since the layer corresponding to the Taobao login interface 3031 is a secure layer, after the Taobao login interface 3021 on the smartphone is displayed on the large-screen device, the Taobao login interface 3021 is displayed in the form of the interface 3031, that is, a black screen interface. The display process containing the secure layer can be specifically referred to Figure 2 for the description, which will not be elaborated here.

[0069] It should be understood that Figure 3 in the scenario shown in , the Taobao login interface 3021 on the smartphone is displayed in the form of a small window. In another scenario, the Taobao login interface 3021 can also be full-screen display. Then, when the Taobao login interface 3021 is full-screen display and is screen-mirrored to the large-screen device, the Taobao login interface 3031 on the large-screen device can also be full-screen display and is a black screen interface.

[0070] In the heterogeneous screen mirroring mode, the first electronic device uses two-way encoding. One-way encoding sends the display interface on the display screen of the first electronic device for display. The other way of encoding sends information such as the standard video stream corresponding to the application interface rendered on the virtual screen to the second electronic device.

[0071] Figure 4 shows the schematic diagram of heterogeneous screen mirroring provided by the embodiment of the present application. Among them, Figure 4 (a) in shows the current output interface 401 of the smartphone, and the content 401 of this interface is the home page interface of the smartphone. In response to the user clicking on the Huawei Video application icon (such as Figure 4 the "Huawei Video" icon shown in (a) of ), after the operation, the mobile phone renders the main interface 401 on the main screen, and renders the Huawei Video interface (such as Figure 4 taking the documentary "Brave Stride" shown in (b) of as an example) 402 interface and the 4021 interface included on the 402 interface on the virtual screen. The interface visible to the user currently is Figure 4 the interface shown in (a) of , Figure 4The interface shown in Figure (b) is the interface displayed on the virtual screen, that is, the user-invisible interface. Based on heterogeneous screen mirroring, as Figure 4 shown in Figure (c), after the large-screen device (i.e., the second electronic device) receives the standard video stream corresponding to the application interface rendered on the virtual screen of the smartphone (i.e., the first electronic device), the large-screen device displays the application interface on the virtual screen of the first electronic device according to the standard video stream. It should be understood that since the video display interface on the virtual screen includes the security layer 4021, the interface 4031 displayed on the main interface 403 of the large-screen device is a black screen interface. Similarly, for the display process of the interface containing the security layer, reference can be specifically made to Figure 2 for the description, which will not be elaborated here.

[0072] It should be understood that Figure 4 in the scenario shown, the Huawei Video interface 4021 on the smartphone is displayed in the form of a small window. In another scenario, the Huawei Video interface 4021 can also be full-screen display. Then, when the Huawei Video interface 4021 is full-screen display and is screen-mirrored to the large-screen device, the Huawei Video interface 4031 on the large-screen device can also be full-screen display and is a black screen interface.

[0073] It can be understood that the homogeneous screen mirroring method and the heterogeneous screen mirroring method each have their own advantages and disadvantages. For example, the homogeneous screen mirroring method can ensure the continuity of the application; while for the heterogeneous screen mirroring method, when switching between different screens, the application needs to be restarted. For example, the A application is displayed on the smartphone side, the B application is created on the virtual screen, and the B application is displayed after being screen-mirrored to the large-screen device. Then, the A application needs to be restarted on the large-screen device to view the A application on the large-screen device. However, the heterogeneous screen mirroring method has better isolation. For example, the heterogeneous screen mirroring method can provide users with independent control screens (i.e., the display screens of the first electronic device and the second electronic device) to process different interfaces.

[0074] In summary, when the layer to be synthesized on the first electronic device includes the security layer, after the synthesized layer is sent to the second electronic device for display, the display area corresponding to the security layer is in a black screen state, resulting in the user being unable to normally view the application content corresponding to the security layer, and even less able to operate the display area corresponding to the security layer using the operating device on the second electronic device, which greatly reduces the user experience.

[0075] In view of this, the present application provides a screen projection display method, by searching for a security layer in the layer to be synthesized on the first electronic device, and changing the first value on the security layer flag to a second value, wherein the first value is used to identify or indicate that the layer is a security layer, and the second value is used to indicate that when synthesizing the layer to be synthesized, the security layer is synthesized normally, so that the synthesized interface can normally display the display area corresponding to the security layer. Then, the first electronic device synthesizes the layer to be synthesized based on the second value, and when the first electronic device projects the synthesized layer containing the security layer to the second electronic device, the display area corresponding to the security layer can be displayed normally on the second electronic device. The method provided in the present application allows the user to obtain the display information of the security layer on the second electronic device, and can also operate on the interface where the security layer is located through the hardware of the second electronic device (such as keyboard, mouse, microphone, speaker, etc.), thereby improving the user experience.

[0076] Exemplarily, the present application can use the WMS in the first electronic device to search for a security layer from the layers to be synthesized, and change the first value corresponding to the security layer to a second value. Then, the Surfaceflinger in the first device uses the indication of the second value to synthesize the security layer normally and send it to the second electronic device for display.

[0077] The screen projection display method provided in the embodiment of the present application can be applied to a communication system composed of a first electronic device and a second electronic device. For example, when the first electronic device projects the screen to the second electronic device, the first electronic device and the second electronic device can be connected wirelessly, for example: the first electronic device and the second electronic device can establish a wireless communication connection by "touch", "scan" (such as scanning a QR code or a barcode), "approach automatic discovery" (such as with the help of Bluetooth or wireless fidelity (WiFi)), etc. Among them, the first electronic device and the second electronic device can follow the wireless transmission protocol and transmit information through a wireless connection transceiver. Among them, the wireless transmission protocol may include but is not limited to the Bluetooth (BT) transmission protocol or the wireless fidelity (WiFi) transmission protocol. For example, the WiFi transmission protocol may be a WiFi P2P transmission protocol. The wireless connection transceiver includes but is not limited to Bluetooth, WiFi and other transceivers. Through wireless pairing, information transmission between the first electronic device and the second electronic device is realized. Among them, the information transmitted between the first electronic device and the second electronic device includes but is not limited to content data (such as standard video streams) and control instructions that need to be displayed.

[0078] The first electronic device and the second electronic device include wireless communication solutions such as wireless local area networks (WLANs), such as wireless fidelity (Wi-Fi) networks, Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared technology (IR).

[0079] Wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), LTE, BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0080] It should be understood that secure screen mirroring in the embodiments of this application refers to the first electronic device and the second electronic device performing screen mirroring through a wired connection. Exemplarily, when both the first electronic device and the second electronic device support the HDCP protocol, it indicates secure screen mirroring between the first electronic device and the second electronic device.

[0081] Alternatively, a wired communication connection is established between the first electronic device and the second electronic device through a video graphics array (VGA), a digital visual interface (DVI), a high definition multimedia interface (HDMI), a data transmission line, or the like.

[0082] The wireless connection between the first electronic device and the second electronic device can be understood as an insecure screen mirroring when the first electronic device mirrors its screen to the second electronic device.

[0083] In the embodiments of the present application, both the first electronic device and the second electronic device include a display screen. The first electronic device and the second electronic device may include, but are not limited to, a smart phone, a netbook, a tablet computer, a smart watch, a smart bracelet, a phone watch, a smart camera, a palm computer, a personal computer (PC), a personal digital assistant (PDA), a portable multimedia player (PMP), an augmented reality (AR) / virtual reality (VR) device, a television, a projection device, or a somatosensory game console in a human-computer interaction scenario, etc. Alternatively, the first electronic device and the second electronic device may also be other types or structures of electronic devices, which are not limited in the present application. Generally, in order to maximize the advantages of the screen mirroring technology, the screen mirroring display method can be applied between a portable device (i.e., the first electronic device) and a large screen device (i.e., the second electronic device). For example, the portable device is a smart phone and the large screen device is a laptop computer. Another example is that the portable device is a tablet computer and the large screen device is a television. Of course, the present application does not limit the specific devices applicable to the screen mirroring display method. As described above, the first electronic device and the second electronic device can be any electronic device that supports screen mirroring, such as a smart phone, a netbook, a tablet computer, a smart watch, a smart bracelet, a phone watch, a smart camera, a palm computer, a PDA, a PMP, an AR / VR device, or a television.

[0084] Exemplarily, Figure 5The structural schematic diagram of an electronic device 100 (such as a first electronic device or a second electronic device) is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0085] It can be understood that the structure schematically shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0086] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modulation and demodulation processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0087] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0088] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the said memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0089] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0090] The I2C interface is a two-way synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be respectively coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: The processor 110 may be coupled to the touch sensor 180K through the I2C interface, enabling the processor 110 to communicate with the touch sensor 180K through the I2C bus interface to implement the touch function of the electronic device 100.

[0091] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple groups of I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to implement the function of answering a call via a Bluetooth headset.

[0092] The PCM interface can also be used for audio communication to sample, quantize, and encode analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to implement the function of answering a call via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0093] The UART interface is a general-purpose serial data bus for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to implement the function of playing music via a Bluetooth headset.

[0094] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0095] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0096] The USB interface 130 is an interface that complies with the USB standard specification. Specifically, it can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other first electronic devices, such as AR devices, etc.

[0097] It can be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only for illustrative purposes and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0098] The charging management module 140 is used to receive a charging input from a charger. Among them, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive the charging input of the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 can receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the first electronic device through the power management module 141.

[0099] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives the inputs from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as the battery capacity, the number of battery cycles, and the battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be provided in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be provided in the same device.

[0100] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0101] Antenna 1 and Antenna 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, Antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0102] The mobile communication module 150 can provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through Antenna 1, filter, amplify and process the received electromagnetic waves, and then transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through Antenna 1 for radiation. In some embodiments, at least some functional modules of the mobile communication module 150 can be provided in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 can be provided in the same device.

[0103] The modulation and demodulation processor can include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, receiver 170B, etc.), or displays an image or video through the display screen 194. In some embodiments, the modulation and demodulation processor can be an independent device. In some other embodiments, the modulation and demodulation processor can be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.

[0104] The wireless communication module 160 may provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 for radiation.

[0105] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, such that electronic device 100 can communicate with a network and other devices via wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TDSCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0106] Electronic device 100 implements a display function via the GPU, display screen 194, and application processor, etc. The GPU is a microprocessor for image processing, and is connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0107] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0108] The electronic device 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0109] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light passes through the lens and is transmitted to the camera photosensitive element. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise, brightness, and skin color of the image through algorithms. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0110] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0111] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0112] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0113] The NPU is a neural-network (NN) computing processor. By drawing on the structure of the biological neural network, such as the transmission mode between human brain neurons, it can quickly process the input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0114] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to achieve the data storage function. For example, files such as music and videos are saved in the external memory card.

[0115] The internal memory 121 can be used to store computer-executable program code, and the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, the image playback function, etc.). The data storage area can store the data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0116] The electronic device 100 can implement audio functions through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and the application processor, etc. For example, music playback, recording, etc.

[0117] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0118] The speaker 170A, also known as the "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or hands-free calls through the speaker 170A.

[0119] The receiver 170B, also known as the "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 100 answers a call or a voice message, the voice can be listened to by holding the receiver 170B close to the ear.

[0120] The microphone 170C, also known as the "microphone" or "transmitter", is used to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can speak close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, and implement a directional recording function, etc.

[0121] The headphone jack 170D is used to connect a wired headphone. The headphone jack 170D can be a USB interface 130, or a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0122] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0123] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0124] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0125] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0126] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the first electronic device, and is applied to applications such as horizontal and vertical screen switching, pedometers, etc.

[0127] A distance sensor 180F is used to measure distance. The electronic device 100 can measure distance through infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0128] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The light-emitting diode may be an infrared light-emitting diode. The electronic device 100 emits infrared light outward through the light-emitting diode. The electronic device 100 uses the photodiode to detect the infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear for a call, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used for automatic unlocking and locking of the holster mode and pocket mode.

[0129] An ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket to prevent accidental touch.

[0130] A fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering of incoming calls, etc.

[0131] A temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 executes a temperature processing strategy using the temperature detected by the temperature sensor 180J. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of the processor near the temperature sensor 180J to reduce power consumption and implement thermal protection. In some other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 caused by low temperature. In some other embodiments, when the temperature is lower than yet another threshold, the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0132] The touch sensor 180K, also known as the "touch panel". The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 together form a touch screen, also known as the "touch display screen". The touch sensor 180K is used to detect touch operations acting thereon or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a different position from that of the display screen 194.

[0133] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals of the vibrating bone mass of the human vocal tract. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulsation signals. In some embodiments, the bone conduction sensor 180M can also be disposed in the earphone to form a bone conduction earphone. The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bone mass acquired by the bone conduction sensor 180M to implement the voice function. The application processor can parse out heart rate information based on the blood pressure pulsation signals acquired by the bone conduction sensor 180M to implement the heart rate detection function.

[0134] The button 190 includes a power-on button, a volume button, etc. The button 190 can be a mechanical button or a touch button. The electronic device 100 can receive button inputs to generate key signal inputs related to the user settings and function control of the electronic device 100.

[0135] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and can also be used for touch vibration feedback. For example, touch operations on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. Touch operations on different regions of the display screen 194 can also cause the motor 191 to correspond to different vibration feedback effects. Different application scenarios (such as time reminder, receiving messages, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0136] The indicator 192 can be an indicator light and can be used to indicate the charging state, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0137] The SIM card interface 195 is used to connect to the SIM card. The SIM card can be inserted into or pulled out from the SIM card interface 195 to achieve contact and separation from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0138] Figure 6 The software structure block diagram of the first electronic device provided by the embodiments of the present application is shown.

[0139] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely the application layer, the application framework layer, the Native layer, the hardware abstraction layer, and the Hardware kernel layer.

[0140] The application layer may include a series of application packages.

[0141] As Figure 6 shown, the applications include system applications and third-party applications. The applications on the electronic device can be system applications (such as the applications installed in the electronic device when the operating system is installed before the electronic device leaves the factory), or third-party applications (such as the applications downloaded and installed by the user through the application store), which are not limited in the embodiments of the present application. For example: the applications may include: camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, game, shopping, travel, instant messaging (such as short message) and other system applications ( Figure 6 not shown in the figure), and the applications may also include third-party applications such as Douyin, WeChat, Taobao, etc. ( Figure 6 not shown in the figure).

[0142] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. As Figure 6As shown, the application framework layer may include a window manager service (WMS), an activity manager service (AMS), and a screen mirroring management module. In some embodiments, the application framework layer may further include a content provider, a view system, a phone manager, a resource manager, a notification manager, etc.( Figure 6 not shown in

[0143] Among them, the WMS carries data and attributes related to windows and is used to manage the states related to windows. For example, it is used to manage window attributes and event dispatching. Among them, managing the window program means orderly outputting to a physical screen or other display devices with the assistance of the application server and the WMS according to the display request of the application. Event dispatching means dispatching user events from keyboards, physical buttons, touchscreens, mice, trackballs (TraceBoll), etc. to the corresponding controls or windows. The window management service can also obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0144] Exemplarily, in the embodiments of the present application, the WMS may traverse the flag bits corresponding to the layers to be synthesized, find the secure layer by looking up the first value of the security flag bit, and then change the first value of the secure layer to a second value.

[0145] For example, the first value may be 0x80, and the first value 0x80 is used to identify that the layer is a secure layer. Then change the first value 0x80 of the secure layer to the second value 0x30. The second value 0x30 is used to indicate that the secure layer is normally synthesized without blackening processing.

[0146] It should be understood that the first value, and / or the second value may also be other identifiers, and the embodiments of the present application do not limit this.

[0147] The AMS is responsible for managing activities, starting, switching, scheduling components in the system, and managing and scheduling application programs. Specifically, data classes are defined in the AMS to store processes, activities, and tasks respectively. Among them, the data class corresponding to a process can include process file information, the memory status information of the process, and the activities, services, etc. contained in the process. Activity information can be stored in the ActivityStack. The Activity Stack is used to uniformly schedule application activities. Specifically, the Activity Stack can store information about all running activities (i.e., the final ArrayList mHistory), such as interface configuration information. For example, the currently running activity can be stored in a new ArrayList. The Activity Stack can also store information about previously run activities, such as interface configuration information. Note that an activity does not correspond to an application program; an Activity Thread corresponds to an application program. Therefore, Android allows multiple application programs to run simultaneously, which actually means allowing multiple Activity Threads to run simultaneously.

[0148] In Android, the basic idea of activity scheduling is as follows: When each application process wants to start a new activity or stop the current activity, it reports to the AMS. The AMS keeps records for all application processes internally. When the AMS receives a start or stop report, it first updates the internal records and then notifies the corresponding client process to run or stop the specified activity. Since the AMS has records of all activities internally, it can schedule these activities and automatically close background activities according to the status of activities and the system memory.

[0149] Content providers are used to store and retrieve data and make this data accessible to application programs. The data can include videos, images, audio, incoming and outgoing calls, browsing history and bookmarks, phone books, etc.

[0150] The view system includes visible controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build application programs. A display interface can consist of one or more views. For example, a display interface including a text message notification icon can include a view for displaying text and a view for displaying pictures.

[0151] The phone manager is used to provide the communication function of the first electronic device. For example, the management of call status (including answering, hanging up, etc.).

[0152] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0153] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages, which can disappear automatically after a short stay without user interaction. For example, the notification manager is used to inform that the download is completed, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as the notification of a background-running application, or a notification that appears on the screen in the form of a dialog window. For example, prompting text information in the status bar, emitting a prompt tone, vibrating the first electronic device, flashing the indicator light, etc.

[0154] The screen mirroring management module is responsible for managing screen mirroring-related matters. For example, transmitting the video stream corresponding to the application interface, interface configuration parameters, etc. Exemplarily, the screen mirroring management module can be Huawei's Assistant or Manager, etc. For example, Assistant can be a module for interacting with other electronic devices (such as the second electronic device) for screen mirroring-related information. For example, Assistant can provide APIs and programming frameworks for the communication between the first electronic device and other electronic devices (such as the second electronic device). Exemplarily, when the screen mirroring device is a computer, Manager can be the computer manager, computer assistant, etc. In some possible embodiments, the screen mirroring management module can not only mirror the programs running on the local device to other devices, but also receive and parse the video stream and related information sent by other devices, so as to display the applications running on other devices on the local device. In some possible embodiments, the screen mirroring management module is segmented and can only implement one of the functions in the above embodiments, so as to meet the requirements of different device forms for the screen mirroring function.

[0155] The native layer includes the interface service (Surfaceflinger) and the hardware renderer (hardware composer, HWC), etc. Each application may correspond to one or more graphical interfaces, and each interface is called a surface. In the implementation of Android, Surfaceflinger is a service that provides the function of system-wide interface composition (surface composer). It can combine 2D and 3D surfaces of various applications. Surfaceflinger can delegate some composition work to HWC to share the workload on OpenGL and GPU.

[0156] In the embodiment of the present application, SurfaceFlinger synthesizes the layers to be synthesized corresponding to the display interface. When SurfaceFlinger traverses the flag bits of each layer, it normally synthesizes the BufferQueue data of the secure layer corresponding to the flag bit of 0x30, so that the display area corresponding to the secure layer can be normally viewed from the user's perspective.

[0157] The hardware abstraction layer (HAL) is used to abstract the hardware. For example, the hardware abstraction layer may include an audio hardware abstraction layer, a Bluetooth hardware abstraction layer, a camera hardware abstraction layer, a hardware renderer abstraction layer, and other hardware device abstraction layers; the hardware renderer abstraction layer may be connected to the hardware renderer algorithm library. After the graphic data is displayed on the device screen, the hardware renderer abstraction layer may call the algorithms in the hardware renderer algorithm library for rendering, and the hardware renderer is the core of all Android graphic rendering.

[0158] The kernel layer is the layer between the hardware and the software. The kernel layer may include a display driver, input / output device drivers (such as a keyboard, a touch screen, headphones, speakers, a microphone, etc.), device nodes, a camera driver, an audio driver, and a sensor driver, etc. The user performs input operations through the input device, and the kernel layer may generate corresponding raw input events according to the input operations and store them in the device nodes.

[0159] It should be noted that Figure 6 Taking only the Android system with a layered architecture as an example, the display process of the application interface in a screen mirroring display scenario is introduced. The present application does not limit the specific architecture of the software system of the electronic device. For the specific introduction of the software systems of other architectures, reference may be made to the conventional technologies. For example, the software system may also include, but is not limited to, Symbian Apple BlackBerry HarmonyOS and other operating systems, which are not limited in the present application. For the sake of easy understanding, the following embodiments of the present application will take an electronic device with the Figure 5 and Figure 6 shown structure as an example, and in combination with the accompanying drawings and application scenarios, specifically elaborate on the screen mirroring display method provided by the embodiments of the present application.

[0160] Figure 7 The schematic flowchart of the screen mirroring display method provided by the embodiment of the present application is shown. As Figure 7 shown, the method 700 includes S710 - S760.

[0161] S710, the first electronic device and the second electronic device establish a screen mirroring connection.

[0162] In the embodiments of the present application, the first electronic device and the second electronic device can be connected in a wired manner or in a wireless manner. The connection manners of the first electronic device and the second electronic device will not be elaborated herein.

[0163] S720, the first electronic device determines a secure layer in the layers to be synthesized of the interface to be cast, and the flag bit of the secure layer is a first value.

[0164] It should be understood that the first electronic device casting the screen to the second electronic device is to project the interface to be displayed created by the first electronic device onto the second electronic device. It should also be understood that the interface to be displayed includes multiple layers to be synthesized.

[0165] When the user performs screen casting, the first electronic device generates multiple layers to be synthesized corresponding to the interface to be displayed in response to the user's operation of starting an application. Before the first electronic device casts the interface to be displayed to the second electronic device, the first electronic device traverses the layers to be synthesized and finds the secure layer from the layers to be synthesized.

[0166] Optionally, in a possible implementation manner, the first electronic device can determine whether there is a secure layer in the layers to be synthesized through the flag bits corresponding to each layer to be synthesized.

[0167] It should be understood that according to the requirements of the Android native system, when the layers to be synthesized contain a secure layer, a security declaration needs to be made for the secure layer. Generally, a security flag bit is marked on the secure layer.

[0168] Exemplarily, the first value on the security flag bit can be 0x80. Of course, the security flag bit can also be marked by other identifiers, or the first value can also be other values. The embodiments of the present application do not limit this.

[0169] It should also be understood that the first value of the security flag bit of the secure layer is used to indicate that when Surfaceflinger synthesizes the layers to be synthesized, the Buffer Queue data of the secure layer is cleared, so that after the layers to be synthesized are synthesized, there is no display information data in the secure layer, and finally the display area corresponding to the secure layer projected onto the second electronic device is black. And, the first value of the security flag bit is also used to indicate that the display area corresponding to the secure layer cannot be screenshot, cannot be recorded, and cannot be multi-task displayed.

[0170] In a possible implementation manner, when the user starts an application, AMS in the first electronic device calls the start activity interface to start an activity, and then, WMS in the first electronic device traverses the layers to be synthesized in the application and determines whether there is a secure layer in the layers to be synthesized.

[0171] Exemplarily, the WMS in the first electronic device may traverse the flag bits of each layer to be synthesized in the application and determine whether the safety flag bit of the safety layer is included in the layers to be synthesized.

[0172] S730, the first electronic device changes the first value to the second value.

[0173] In the embodiment of the present application, after the first electronic device finds the safety layer, it may change the safety flag bit corresponding to the safety layer, so that the safety layer can be normally synthesized on the first electronic device. Finally, after the interface to be displayed is cast to the second electronic device, the display area corresponding to the safety layer can be normally displayed on the second electronic device.

[0174] In some possible implementation manners, the first electronic device may change the first value corresponding to the safety layer to the second value.

[0175] It should be understood that the first value is used to indicate that the layer is a safety layer. Then, the first electronic device performs safety processing when synthesizing the safety layer, that is, clears the information of the display area corresponding to the safety layer. And, the first electronic device ensures that the safety layer cannot be screenshot, cannot be recorded, and cannot be multi-task displayed. The second value is used to indicate that when the first electronic device synthesizes the layers to be synthesized, the display data corresponding to the safety layer in the layers to be synthesized is normally displayed.

[0176] It should also be understood that the second value is only used as an indication identifier to indicate that when the first electronic device synthesizes the layers to be synthesized, the display area information corresponding to the safety layer is normally displayed, and at the same time, it is ensured that the safety layer cannot be screenshot, cannot be recorded, and cannot be multi-task displayed. That is, regardless of whether the value of the safety flag bit corresponding to the safety layer is the first value or the second value, the safety layer cannot be screenshot, cannot be recorded, and cannot be multi-task displayed.

[0177] As a possible implementation manner, the first value is used to indicate that when Surfaceflinger in the first electronic device synthesizes the layers to be synthesized, the Buffer Queue data in the safety layer is cleared, so that after the layers to be synthesized are synthesized, there is no display information data in the safety layer, and finally the display area corresponding to the safety layer projected onto the second electronic device is black. The second value is used to indicate that when Surfaceflinger in the first electronic device synthesizes the layers to be synthesized, the Buffer Queue data in the safety layer is normally displayed, so that after the interface to be displayed is synthesized, the display information data in the safety layer is retained, and finally the display area corresponding to the safety layer projected onto the second electronic device is normally displayed.

[0178] Exemplarily, the first value may be the security flag bit 0x80 of the security layer in step S720, and the second value may be 0x30. Of course, the second value may also be other values, and the embodiments of the present application do not limit this.

[0179] In this embodiment, by searching for all the layer flag bits to be synthesized of the interface to be displayed in the first electronic device, the security layer in the layers to be synthesized is found. Then, the first value of the security flag bit corresponding to the security layer is changed to the second value, so that when the first electronic device synthesizes all the layers to be synthesized, the security layer is synthesized normally, and finally, after being projected onto the second electronic device, the second electronic device can normally display the area corresponding to the security layer. This implementation method of changing the value of the security flag bit can only change the synthesis method of the security layer without changing other attributes of the security layer. For example, the display area corresponding to the security layer cannot be screenshot, cannot be recorded, and cannot be multitasked. Thus, while ensuring that the security layer can be normally displayed, the security of the security layer is guaranteed.

[0180] S740, the first electronic device synthesizes the layers to be synthesized according to the second value, so that the display area corresponding to the security layer is normally displayed.

[0181] When the first electronic device detects that the security flag bit corresponding to the security layer in the layers to be synthesized is the second value, when the first electronic device synthesizes the layers to be synthesized, it normally synthesizes the security layer on the layers to be synthesized, that is, retains the display information of the security layer, so that the display area corresponding to the security layer on the layers to be synthesized projected onto the second electronic device can be normally displayed.

[0182] As a possible implementation method, when Surfaceflinger on the first electronic device synthesizes the layers to be synthesized, it traverses the layers to be synthesized. When it traverses to the security flag bit of the security layer being the second value, it normally synthesizes the security layer on the layers to be synthesized.

[0183] Specifically, when Surfaceflinger on the first electronic device synthesizes the layers to be synthesized, Surfaceflinger first traverses the layers to be synthesized. When it finds that the security flag bit of the security layer traversed is the second value, Surfaceflinger will normally synthesize all the layers according to the Buffer Queue data corresponding to the security layer, rather than as Figure 2As shown, when SurfaceFlinger synthesizes the layers to be synthesized, it clears the Buffer Queue data corresponding to the secure layer. It should be understood that after SurfaceFlinger normally synthesizes the secure layer according to the indication of the second value, after the second electronic device receives the synthesized layer from the first electronic device, it can normally display the display area corresponding to the secure layer only after local rendering.

[0184] It should be noted that SurfaceFlinger can synthesize all layers, or it can also delegate some synthesis work to HWC for layer synthesis. For example, SurfaceFlinger can provide a complete list of all layers to HWC, and let HWC decide which layers to synthesize according to its hardware capabilities.

[0185] In S750, the first electronic device sends screen mirroring data to the second electronic device, and the screen mirroring data includes the synthesized layer.

[0186] The first electronic device sends the screen mirroring data to the second electronic device for the second electronic device to display the screen mirroring interface corresponding to the interface to be displayed. In the embodiments of the present application, the screen mirroring data includes the synthesized layer.

[0187] In a possible implementation manner, after SurfaceFlinger in the first electronic device synthesizes the layers to be synthesized, it can send the synthesized layer to the HWC of the second electronic device, and after the HWC of the second electronic device renders it, the second electronic device displays the screen mirroring interface corresponding to the interface to be displayed.

[0188] Optionally, when the first electronic device sends the synthesized layer, it can encode the synthesized layer into a video stream, and according to the Real-Time Transport Protocol (RTP) or Real-Time Control Protocol (RTCP), pack the encoded and compressed video stream data, and then send the packed video stream to the second electronic device.

[0189] Specifically, after SurfaceFlinger in the first electronic device synthesizes all the layers to be synthesized, it sends the synthesized layer to the encoder to generate a standard video stream. Then, the first electronic device performs RTP or RTCP packing on the obtained standard video stream. Finally, the first electronic device sends the packed data to the second electronic device through TCP or UDP.

[0190] In this embodiment, since the first electronic device encodes and compresses the synthesized layer, and performs RTP or RTCP packing on the encoded layer, and then sends the packed layer to the second electronic device, this can not only improve the security of the layer data, but also reduce the data transmission volume and improve the data transmission efficiency.

[0191] On the S760, the second electronic device displays an interface to be cast.

[0192] In an embodiment of the present application, after receiving the video stream of the synthesized layers sent by the first electronic device, the second electronic device needs to parse the video stream to obtain all the layers to be displayed.

[0193] Exemplarily, the second electronic device can unpack the video stream of the synthesized layers according to RTP or RTCP, and decode the unpacked layers to obtain the display information of all the layers.

[0194] In a possible implementation manner, if the first electronic device uses RTP to pack the encoded layers, correspondingly, the second electronic device will use RTP to unpack the received video stream of the synthesized layers. If the first electronic device uses RTCP to pack the encoded layers, correspondingly, the second electronic device will use RTCP to unpack the received video stream of the synthesized layers. Then, the unpacked layers are sent to a decoder to decode the unpacked layers.

[0195] Finally, the HWC in the second electronic device outputs the video stream data of all the decoded layers to the frame buffer (FrameBuffer), and then outputs it to the second electronic device for display. Alternatively, the HWC directly outputs all the synthesized layer data to the display screen on the second electronic device for display.

[0196] Optionally, in a possible implementation manner, the first electronic device can also send the layers to be synthesized to the second electronic device. The second electronic device normally synthesizes the layers to be synthesized corresponding to the interface to be displayed according to the second value.

[0197] Specifically, the first electronic device can encode the layers to be synthesized into a video stream, pack the encoded and compressed video stream data according to the real-time transport protocol RTP or RTCP control protocol, and then send the packed video stream to the second electronic device.

[0198] After receiving the video stream of the layers to be synthesized sent by the first electronic device, the second electronic device needs to parse the video stream to obtain all the layers to be synthesized. Then, the layers to be synthesized are synthesized to obtain the synthesized layers.

[0199] Finally, the second electronic device renders the synthesized layer in the local HWC and directly outputs the rendered layer to the display screen on the second electronic device for display. Alternatively, the HWC outputs the rendered layer to the Frame Buffer, and then outputs it to the second electronic device for display.

[0200] Optionally, as another possible implementation, after the Surfaceflinger on the first electronic device synthesizes all the layers, the remaining layer synthesis, optimization, and rendering can also be completed through the local HWC.

[0201] After that, the HWC on the first electronic device sends the optimized and rendered layer to the Surfaceflinger on the first electronic device, and then the Surfaceflinger sends it to the WMS. The WMS sends the to-be-synthesized layer after synthesis to Figure 4 the screen mirroring management module of the first electronic device as shown. The screen mirroring management module of the first electronic device sends the to-be-synthesized layer after synthesis directly to the second electronic device (such as the screen mirroring management module of the second electronic device) for display.

[0202] In this implementation, the first electronic device does not need to encode, compress, and package the synthesized layer and send it to the second electronic device, and then perform rendering and display after decoding and decompressing by the second electronic device, which improves the efficiency of screen mirroring display.

[0203] The screen mirroring display method provided by the embodiments of the present application searches for a secure layer in the to-be-synthesized layers on the first electronic device, and changes the first value of the security flag bit corresponding to the secure layer to a second value. The first value is used to identify or indicate that the layer is a secure layer, and the second value is used to indicate that when synthesizing the to-be-synthesized layers, the secure layer is normally synthesized, so that the corresponding display area of the secure layer can be normally displayed in the synthesized interface. Then, the first electronic device synthesizes the to-be-synthesized layers based on the second value. When the first electronic device screens the synthesized layers including the secure layer to the second electronic device, the corresponding display area of the secure layer can be normally displayed on the second electronic device. The method provided by the present application enables the user to obtain the display information of the secure layer on the second electronic device, and can also operate on the interface where the secure layer is located through the hardware of the second electronic device (such as keyboard, mouse, microphone, speaker, etc.), thereby improving the user experience.

[0204] Optionally, before the first electronic device executes step S720, the first electronic device can determine whether the screen mirroring between the first electronic device and the second electronic device is non-secure.

[0205] It should be understood that in the related art, when the screen mirroring between the first electronic device and the second electronic device is secure, and the layer to be synthesized for screen mirroring from the first electronic device to the second electronic device includes a secure layer, the display area corresponding to the secure layer is normally displayed on the second electronic device. When the screen mirroring between the first electronic device and the second electronic device is non-secure, and the layer to be synthesized for screen mirroring from the first electronic device to the second electronic device includes a secure layer, the display area corresponding to the secure layer displayed on the second electronic device is black.

[0206] Therefore, in the embodiments of the present application, it can be determined whether the screen mirroring between the first electronic device and the second electronic device is secure. If the screen mirroring between the first electronic device and the second electronic device is non-secure, the screen mirroring display method provided in the embodiments of the present application can be executed.

[0207] In a possible implementation manner, when the first electronic device and the second electronic device are wirelessly connected, it indicates that the screen mirroring between the first electronic device and the second electronic device is non-secure.

[0208] Exemplarily, the first electronic device and the second electronic device can determine whether they are wirelessly connected by determining whether they support the HDCP protocol. When the first electronic device and the second electronic device are wirelessly connected, it indicates that the screen mirroring between the first electronic device and the second electronic device is non-secure.

[0209] Specifically, when both the first electronic device and the second electronic device support the HDCP protocol, it indicates that the first electronic device and the second electronic device are wired-connected, and it indicates that the screen mirroring between the first electronic device and the second electronic device is secure; when at least one end of the first electronic device and the second electronic device does not support the HDCP protocol, it indicates that the first electronic device and the second electronic device are wirelessly connected, and it indicates that the screen mirroring between the first electronic device and the second electronic device is non-secure.

[0210] In the embodiments of the present application, when the first electronic device performs screen mirroring to the second electronic device, the first electronic device and the second electronic device respectively determine whether the local end supports the HDCP protocol. Then, the second electronic device sends the determination result to the first electronic device. When the first electronic device determines that at least one end of the first electronic device and the second electronic device does not support the HDCP protocol, it indicates that the screen mirroring between the first electronic device and the second electronic device is non-secure.

[0211] When it is secure screen mirroring between the first electronic device and the second electronic device, the flag bit of the interface to be screen mirrored created by the first electronic device is secure, and this flag bit is used to indicate that the interface to be screen mirrored is a secure interface, which means it is secure screen mirroring between the first electronic device and the second electronic device; when the flag bit of the interface to be screen mirrored is not secure, this flag bit is used to indicate that the interface to be screen mirrored is a non-secure interface, indicating that it is non-secure screen mirroring between the first electronic device and the second electronic device.

[0212] It should be understood that in the embodiments of the present application, the interface to be screen mirrored can also be understood as the display to be screen mirrored.

[0213] Of course, it is also possible to determine whether it is secure screen mirroring between the first electronic device and the second electronic device through other means, and the embodiments of the present application do not limit this.

[0214] The above method for screen mirroring display provided by the present application can be applied to scenarios of homologous screen mirroring or heterologous screen mirroring. The following takes the first electronic device as a smart phone and the second electronic device as a large screen device as an example to specifically illustrate the application scenarios of the above screen mirroring display method. Figure 8 The schematic diagram of the display interface of the secure layer under homologous screen mirroring provided by the embodiments of the present application is shown. Among them, Figure 8 Figure (a) in it shows the current output interface 801 of the smart phone, and the content 801 of this interface is the home page interface of the smart phone. In response to the user's operation of clicking on the Taobao application icon (such as the "Taobao" icon shown in Figure (a) in Figure 8 ), as shown in Figure (b) in Figure 8 , the smart phone renders the home page interface 802 and the Taobao login interface 8021 on the home page interface 802 together on the main screen. In response to the user's screen mirroring operation, the WMS on the smart phone detects the flag bits of all layers on the interface 802. When it detects that the flag bit of the secure layer 8021 on the interface 802 is 0x80, it means that the Taobao login interface 8021 is a secure layer, and then changes the secure flag bit of this secure layer to 0x30. This 0x30 means that the secure layer is normally synthesized during composition. Then, when Surfaceflinger on the smart phone performs multi-layer composition, it traverses the flag bits of all layers. When it detects that the flag bit of a layer is 0x30, it normally synthesizes this layer. Finally, Surfaceflinger encodes the data corresponding to the synthesized layer into a standard video and sends it to the large screen device.

[0215] Based on homologous screen mirroring, after the large screen device receives the standard video stream corresponding to the Surface encoding of the application interface rendered on the main screen of the smart phone, the large screen device displays the home page interface 803 and the Taobao login interface 8031 included on the home page interface 803 according to this standard video stream. AsFigure 8 As shown in Figure (c) of , although the layer corresponding to the Taobao login interface is a secure layer, after the Taobao login interface 8031 is displayed on a large-screen device, it can still be normally displayed, and this secure layer cannot be screenshot, cannot be screen-recorded, and cannot be multi-task displayed.

[0216] It should be understood that Figure 8 In the scenario shown, the Taobao login interface 8021 on the smart phone is displayed in the form of a small window. In another scenario, the Taobao login interface 8021 can also be full-screen displayed. Then, when the Taobao login interface 8021 is full-screen displayed and cast to a large-screen device, the Taobao login interface 8031 on the large-screen device can also be full-screen displayed. Although the layer corresponding to the Taobao login interface is a secure layer, after the Taobao login interface 8031 is displayed on the large-screen device, it can still be normally displayed.

[0217] Figure 9 Figure shows a schematic diagram of the display interface of the secure layer under heterogeneous screen mirroring provided by the embodiment of the present application. Among them, Figure 9 Figure (a) of shows the current output interface 901 of the smart phone, and the interface content 901 is the home page interface of the smart phone. In response to the user's operation of clicking on the Huawei Video application icon (such as Figure 9 the "Huawei Video" icon shown in Figure (a) of ), the mobile phone renders the home page 901 of the mobile phone on the main screen 902 and renders the Huawei Video interface (taking the documentary "Brave Step" shown in Figure (b) of Figure 9 as an example) 9021 on the virtual screen. In response to the user's screen mirroring operation, the WMS on the smart phone detects the layer flag bits on the interface 902. When it detects that the flag bit of the secure layer 9021 on the interface 902 is 0x80, it indicates that the video playback interface 9021 is a secure layer, and then changes the security flag bit of this secure layer to 0x30. This 0x30 indicates that the secure layer is normally synthesized during composition. Then, when the Surfaceflinger on the smart phone performs multi-layer composition, it traverses the flag bits of all layers. When it detects that the flag bit of a layer is 0x30, it normally synthesizes this layer. Finally, the Surfaceflinger on the smart phone encodes the data corresponding to the synthesized layer into a standard video stream and sends it to the large-screen device.

[0218] Based on heterogeneous screen mirroring, such as Figure 9 ​As shown in Figure (c), after the large-screen device (i.e., the second electronic device) receives the standard video stream corresponding to the application interface rendered on the virtual screen from the smart phone (i.e., the first electronic device), the large-screen device displays the video application interface according to the standard video stream. It should be noted that although the video display interface on the virtual screen includes a security layer, the display interface 9031 corresponding to the security layer on the main interface 903 of the large-screen device is normally displayed, and this security layer cannot be screenshot, cannot be recorded, and cannot be multi-task displayed.

[0219] It should be understood that Figure 9 In the scenario shown, the Huawei Video interface 9021 on the smart phone is displayed in the form of a small window. In another scenario, the Huawei Video interface 9021 can also be full-screen displayed. Then, when the Huawei Video interface 9021 is full-screen displayed and cast to the large-screen device, the Huawei Video interface 9031 on the large-screen device can also be full-screen displayed. Although the layer corresponding to the Huawei Video interface is a security layer, after the Huawei Video interface 9031 is displayed on the large-screen device, it can still be normally displayed.

[0220] The embodiment of the present application further provides a chip system, as Figure 10 shown. This chip system includes at least one processor 1001 and at least one interface circuit 1002. The processor 1001 and the interface circuit 1002 can be interconnected through a line. For example, the interface circuit 1002 can be used to receive signals from other devices (such as the memory of any of the above-mentioned electronic devices). For another example, the interface circuit 1002 can be used to send signals to other devices (such as the processor 1001). Exemplarily, the interface circuit 1002 can read the instructions stored in the memory and send the instructions to the processor 1001. When the instructions are executed by the processor 1001, the electronic device can execute each step executed by any of the electronic devices in the above embodiments. Of course, this chip system can also include other discrete devices, and the embodiments of the present application do not make specific limitations on this.

[0221] It should also be understood that the division of units in the above device is only a division of logical functions. In actual implementation, it can be fully or partially integrated into a physical entity, or physically separated. And the units in the device can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; or some units can be implemented in the form of software called by processing elements, and some units can be implemented in the form of hardware. For example, each unit can be a separately established processing element, or can be integrated in a certain chip of the device. In addition, it can also be stored in the memory in the form of a program, and the function of the unit is called and executed by a certain processing element of the device. Here, this processing element can also be called a processor, which can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above units can be implemented by the integrated logic circuit of the hardware in the processor element or in the form of software called by the processing element. In one example, the units in any of the above devices can be one or more integrated circuits configured to implement the above method, for example: one or more application specific integrated circuits (ASICs), or, one or more digital signal processors (DSPs), or, one or more field programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms. Again, for example, when the units in the device can be implemented in the form of a processing element scheduling program, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call programs. Again, for example, these units can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0222] The embodiment of the present application also provides a device, which is included in an electronic device (such as the first electronic device or the second electronic device), and the device has the function of implementing the behavior of the electronic device in any of the above embodiments. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes at least one module or unit corresponding to the above function.

[0223] The present application also provides an electronic device (such as the first electronic device or the second electronic device), and the electronic device includes the device provided in the embodiment of the present application described above.

[0224] The embodiments of the present application also provide a computer-readable storage medium for storing computer program code. The computer program includes instructions for performing the steps of the electronic device (such as the first electronic device or the second electronic device) executing the display interface in any of the embodiments provided in the embodiments of the present application. The readable medium may be a read-only memory (ROM) or a random access memory (RAM), and the embodiments of the present application do not limit this.

[0225] The present application also provides a computer program product. The computer program product includes instructions that, when executed, cause the electronic device to perform the steps of the electronic device executing or displaying the interface in any of the above embodiments.

[0226] The embodiments of the present application also provide a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin, or a circuit, etc. The processing unit can execute computer instructions to enable the electronic device to perform any of the screen mirroring display methods provided in the embodiments of the present application.

[0227] Optionally, the computer instructions are stored in a storage unit.

[0228] Optionally, the storage unit is a storage unit within the chip, such as a register or a cache, etc. The storage unit can also be a storage unit outside the chip within the terminal, such as a ROM or other types of static storage devices that can store static information and instructions, a random RAM, etc. Among them, the processor mentioned anywhere above can be a CPU, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned screen mirroring display method of the electronic device. The processing unit and the storage unit can be decoupled and set on different physical devices, and are connected by wired or wireless means to achieve the respective functions of the processing unit and the storage unit, so as to support the system chip to implement various functions in the above embodiments. Or, the processing unit and the memory can also be coupled on the same device.

[0229] Among them, the electronic device, device, computer-readable storage medium, computer program product, or chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be elaborated here.

[0230] An embodiment of the present application further provides a graphical user interface on an electronic device. The electronic device has a display screen, a camera, a memory, and one or more processors. The one or more processors are configured to execute one or more computer programs stored in the memory. The graphical user interface includes the graphical user interface displayed when the electronic device executes the steps performed by the electronic device in any of the above embodiments.

[0231] It can be understood that, in order to implement the above functions, the above-mentioned electronic devices, etc., include the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed in this article, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving the hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0232] The embodiments of the present application can divide the above-mentioned electronic devices, etc. into functional modules according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0233] Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be repeated here.

[0234] In each embodiment of the present application, each functional unit can be integrated in one processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0235] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disks, read-only memories, random access memories, magnetic disks, or optical discs.

[0236] As described above, the above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for screen mirroring display, characterized in that, Applied to a first electronic device, the first electronic device establishes a screen mirroring connection with a second electronic device, and the method includes: In response to a user operation, generating a plurality of layers to be synthesized corresponding to the screen mirroring interface, where the plurality of layers to be synthesized includes a security layer; Changing the flag bit of the security layer in the plurality of layers to be synthesized to a second value, where the second value is used to indicate that when the first electronic device synthesizes the layers to be synthesized, the display data corresponding to the security layer is normally displayed; Synthesizing the layers to be synthesized according to the second value; Sending screen mirroring data to the second electronic device, where the screen mirroring data includes the synthesized layers.

2. The method according to claim 1, wherein After generating a plurality of layers to be synthesized corresponding to the screen mirroring interface and before changing the flag bit of the security layer to the second value, the method further includes: Determining the security layer in the plurality of layers to be synthesized, where the flag bit of the security layer is a first value, and the first value is used to indicate that when synthesizing the layers to be synthesized, the data in the security layer is cleared.

3. The method according to claim 2, wherein Before determining the security layer in the layers to be synthesized of the screen mirroring interface, the method further includes: Determining that the screen mirroring between the first electronic device and the second electronic device is non-secure.

4. The method according to any one of claims 1-3, characterized in that, In the synthesized layers, the display area corresponding to the security layer cannot be screenshot, cannot be recorded, and cannot be multitasked.

5. The method according to claim 2 or 3, characterized in that, The window management service WMS in the first electronic device changes the first value to the second value.

6. The method according to any one of claims 1-5, characterized in that The interface synthesis system surfaceflinger in the first electronic device synthesizes the layers to be synthesized according to the second value.

7. The method according to any one of claims 2-6, characterized in that, The first value includes: 0x80.

8. The method according to any one of claims 1 to 7, characterized in that The second value includes: 0x30.

9. The method according to any one of claims 3-8, characterized in that When the first electronic device and the second electronic device are wirelessly connected, the first electronic device is in a non-secure screen mirroring state.

10. The method according to any one of claims 3-9, characterized in that, When the flag bit in the screen mirroring interface is used to indicate that the screen mirroring interface is a non-secure interface, the first electronic device is in a non-secure screen mirroring state.

11. The method according to any one of claims 1 to 10, characterized in that, The first electronic device performs screen mirroring to the second electronic device in a homologous screen mirroring manner or a heterologous screen mirroring manner.

12. A system for screen mirroring display, characterized in that, The system includes a first electronic device and a second electronic device, and the first electronic device is used to perform screen mirroring to the second electronic device; The first electronic device, in response to a user operation, generates a plurality of layers to be synthesized corresponding to the screen mirroring interface, where the plurality of layers to be synthesized includes a security layer; The first electronic device changes the flag bit of the security layer in the plurality of layers to be synthesized to a second value, where the second value is used to indicate that when the first electronic device synthesizes the layers to be synthesized, the display data corresponding to the security layer is normally displayed; The first electronic device is further used to synthesize the layers to be synthesized according to the second value; The first electronic device is further used to send screen mirroring data to the second electronic device, where the screen mirroring data includes the synthesized layers.

13. The system according to claim 12, wherein After generating a plurality of layers to be synthesized corresponding to the screen mirroring interface and before changing the flag bit of the security layer to the second value, the first electronic device is further used to: Determine the secure layer among the multiple layers to be synthesized, where the flag bit of the secure layer is the first value, and the first value is used to indicate that when synthesizing the layers to be synthesized, the data in the secure layer is cleared.

14. The system according to claim 13, wherein Before the first electronic device determines the secure layer among the layers to be synthesized in the interface to be projected, the first electronic device is further configured to determine that the projection between the first electronic device and the second electronic device is an insecure projection.

15. The system according to any one of claims 12-14, characterized in that, In the synthesized layer, the display area corresponding to the secure layer cannot be captured by screenshot, cannot be recorded by screen recording, and cannot be displayed in multitasking.

16. The system according to claim 13 or 14, characterized in that, The WMS in the first electronic device is used to change the first value to the second value.

17. The system according to any one of claims 12-15, characterized in that, The surfaceflinger in the first electronic device is used to synthesize the layers to be synthesized according to the second value.

18. The system according to any one of claims 13-17, wherein The first value includes: 0x80.

19. The system according to any one of claims 12-18, characterized in that, The second value includes: 0x30.

20. The system according to any one of claims 13-19, characterized in that, When the first electronic device and the second electronic device are wirelessly connected, the first electronic device is in an insecure projection state.

21. The system according to any one of claims 13-20, characterized in that, When the flag bit in the interface to be projected is used to indicate that the interface to be projected is an insecure interface, the first electronic device is in an insecure projection state.

22. The system according to any one of claims 13-21, characterized in that The first electronic device projects the screen to the second electronic device in a way of homologous screen projection or heterologous screen projection.

23. An electronic device, characterized in that, The electronic device includes a processor and a memory. The memory is used to store instructions, and the processor is used to read the instructions to execute the method according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 11.

25. A chip, characterized in that, Comprising: A processor, configured to call and run a computer program from a memory, so that a communication device installed with the chip executes the method according to any one of claims 1 to 11.

Images