Screen projection display method, system and electronic device

Abstract

The present application provides a method, system and electronic device for screen projection display, the method is applied to a first electronic device, the first electronic device and a second electronic device establish a screen projection connection, the method includes: determining a security layer in a layer to be synthesized of a screen projection interface, the flag of the security layer being a first value; changing the first value to a second value, synthesizing the layer to be synthesized according to the second value, so that the display area corresponding to the synthesized security layer is displayed normally; sending screen projection data to the second electronic device, the screen projection data including the synthesized layer. By changing the value in the flag bit of the security layer, the present application can only change the synthesis method of the security layer without changing the other properties of the security layer, so that the security of the security layer can be guaranteed while the display area corresponding to the security layer can be displayed normally.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and more specifically, to a method, system and electronic device for screen projection display. Background Art

[0002] With the development of application display technology, more and more electronic devices support screen projection technology. Among them, screen projection technology is to project the application interface started on an electronic device with a smaller display screen (such as a first electronic device) to another electronic device with a larger display screen (such as a second electronic device), so as to achieve a better viewing effect, and the application interface can be operated by the operating device on the second electronic device.

[0003] However, when the application interface launched on the first electronic device includes a security layer and the application interface is projected onto the second electronic device, the display area corresponding to the security layer is black, causing the user to be unable to view the information corresponding to the security layer, let alone operate the display interface corresponding to the security layer, greatly reducing the user experience. Summary of the invention

[0004] The present application provides a method, system and electronic device for screen projection display. By changing the value of the flag bit of the security layer in the layer to be synthesized, the security layer can be synthesized normally, thereby solving the problem in the related art that the user cannot identify the display area corresponding to the security layer, and improving the user experience.

[0005] In a first aspect, a method for screen projection display is provided, the method is applied to a first electronic device, the first electronic device and a second electronic device establish a screen projection connection, and the execution subject of the method can be the first electronic device or a chip installed in the first electronic device. The method includes: determining a security layer in a layer to be synthesized of a screen projection interface, the flag bit of the security layer is a first value; changing the first value to a second value; synthesizing the layer to be synthesized according to the second value so that the display area corresponding to the security layer is displayed normally; sending screen projection data to the second electronic device, the screen projection data including the synthesized layer.

[0006] In the method of screen projection display provided in the first aspect, the first electronic device searches for the security layer in the layer to be synthesized corresponding to the interface to be displayed, and then changes the first value of the flag bit of the security layer to the second value, so that the first electronic device synthesizes the security layer normally when synthesizing the layer to be synthesized. The display area corresponding to the security layer synthesized in this way can be displayed normally. By changing the flag bit value, the method can only change the synthesis method of the security layer without changing the other properties of the security layer. For example, the display area corresponding to the security layer cannot be screenshot, cannot record the screen, and cannot display multiple tasks. Thereby ensuring the security of the security layer under the premise of normal display.

[0007] In a possible implementation of the first aspect, before determining the security layer in the layer to be synthesized of the screen projection interface, the method further includes: determining that the screen projection between the first electronic device and the second electronic device is non-secure. In the related art, when the screen projection between the first electronic device and the second electronic device is non-secure, the display area corresponding to the security layer is a black screen. Therefore, when it is determined that the screen projection between the first electronic device and the second electronic device is non-secure, the screen projection display method provided by the present application can be used to view the display area corresponding to the security layer while ensuring the security of the layer.

[0008] For example, when the first electronic device and the second electronic device are wirelessly connected, the first electronic device is in a non-secure screen projection state.

[0009] For another example, when a flag bit in the screen to be projected interface is used to indicate that the screen to be projected interface is a non-secure interface, the first electronic device is in a non-secure screen projection state.

[0010] In a possible implementation of the first aspect, in the synthesized layer, the display area corresponding to the security layer cannot be screenshotted, cannot be recorded, and cannot be displayed in multiple tasks. This implementation ensures the security of the security layer while ensuring the normal display of the security layer.

[0011] In a possible implementation manner of the first aspect, the window management service WMS in the first electronic device may change the first value to a second value.

[0012] In a possible implementation of the first aspect, the interface synthesis system surfaceflinger in the first electronic device can synthesize the layer to be synthesized according to the second value.

[0013] In a possible implementation manner of the first aspect, the first value may be 0×80, and the second value may be 0×30.

[0014] In a possible implementation of the first aspect, the first electronic device projects the screen to the second electronic device in a same-source screen projection manner or a different-source screen projection manner.

[0015] In a second aspect, a screen projection display system is provided, the system comprising a first electronic device and a second electronic device, the first electronic device being used to project the screen to the second electronic device; the first electronic device being used to determine a security layer in a layer to be synthesized of a screen projection interface, the flag bit of the security layer being a first numerical value; the first electronic device being also used to change the first numerical value to a second numerical value; the first electronic device being also used to synthesize the layer to be synthesized according to the second numerical value, so that a display area corresponding to the synthesized security layer is displayed normally; the first electronic device being also used to synthesize the layer to be synthesized according to the second numerical value, so that a display area corresponding to the security layer is displayed normally; the first electronic device being also used to send projection data to the second electronic device, the projection data including the synthesized layer.

[0016] The second aspect provides a screen projection display system, which searches for a security layer in all layers to be synthesized of the interface to be displayed in the first electronic device, and then changes the first value of the flag of the security layer to the second value, so that the first electronic device synthesizes the security layer normally when synthesizing all layers to be synthesized, and finally after the screen is projected to the second electronic device, the second electronic device can normally display the area corresponding to the security layer. This method of changing the flag value can only change the synthesis method of the security layer without changing the other properties of the security layer, for example, the display area corresponding to the security layer cannot be screenshotted, cannot be recorded, and cannot be multi-tasking displayed. Thereby ensuring the security of the security layer under the premise of normal display.

[0017] In a possible implementation of the second aspect, before the first electronic device determines the security layer in the layer to be synthesized of the screen projection interface, the first electronic device is also used to determine that the screen projection between the first electronic device and the second electronic device is non-secure. In the related art, when the screen projection between the first electronic device and the second electronic device is non-secure, the display area corresponding to the security layer is a black screen. Therefore, when it is determined that the screen projection between the first electronic device and the second electronic device is non-secure, the screen projection display method provided by the present application can be used to view the display area corresponding to the security layer while ensuring the security of the layer.

[0018] For example, when the first electronic device and the second electronic device are wirelessly connected, the first electronic device is in a non-secure screen projection state.

[0019] For another example, when a flag bit in the screen to be projected interface is used to indicate that the screen to be projected interface is a non-secure interface, the first electronic device is in a non-secure screen projection state.

[0020] In a possible implementation of the second aspect, in the synthesized layer, the display area corresponding to the security layer cannot be screenshotted, recorded, or displayed in multiple tasks. This implementation ensures the security of the security layer while ensuring the normal display of the security layer.

[0021] In a possible implementation manner of the second aspect, the window management service WMS in the first electronic device may change the first value to the second value.

[0022] In a possible implementation of the second aspect, the interface synthesis system surfaceflinger in the first electronic device can synthesize the layer to be synthesized according to the second value.

[0023] In a possible implementation manner of the second aspect, the first value may be 0×80, and the second value may be 0×30.

[0024] In a possible implementation of the second aspect, the first electronic device projects the screen to the second electronic device in a same-source screen projection manner or a different-source screen projection manner.

[0025] In a possible implementation of the second aspect, the first electronic device is used to encode the synthesized layer to obtain the encoded layer; the first electronic device is also used to package the encoded layer according to the real-time transport protocol RTP or the RTP control protocol RTCP; the first electronic device is used to send the packaged layer to the second electronic device. In this implementation, the first electronic device encodes and compresses all layers of the layer to be synthesized, and performs RTP or RTCP packaging processing on the encoded layer, thereby sending the packaged layer to the second electronic device, thereby not only improving the security of the layer data, but also reducing the amount of data transmission and improving the efficiency of data transmission.

[0026] According to a third aspect, a communication device is provided, which includes a unit for executing each step in the above first aspect or any possible implementation of the first aspect.

[0027] In a fourth aspect, a communication device is provided, which includes at least one processor and a memory, the processor and the memory are coupled, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method in the above first aspect or any possible implementation of the first aspect is executed.

[0028] In a fifth aspect, a communication device is provided, comprising at least one processor and an interface circuit, wherein the at least one processor is used to execute the method in the above first aspect or any possible implementation of the first aspect.

[0029] In a sixth aspect, an electronic device is provided, which includes the communication device provided in the second aspect, or the electronic device includes the communication device provided in the third aspect, or the electronic device includes the communication device provided in the fourth aspect.

[0030] In a seventh aspect, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, it is used to execute the method in the above first aspect or any possible implementation of the first aspect.

[0031] In an eighth aspect, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed, it is used to execute the method in the above first aspect or any possible implementation of the first aspect.

[0032] In a ninth aspect, a chip is provided, comprising: a processor for calling and running a computer program from a memory, so that a communication device equipped with the chip executes a method for executing the above first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic flow chart of an electronic device displaying interface information in a screen projection scenario in the related art is shown;

[0034] Figure 2 A schematic diagram showing a process of synthesizing layers to be synthesized on a first electronic device;

[0035] Figure 3 A schematic diagram of same-source screen projection provided in an embodiment of the present application is shown;

[0036] Figure 4 A schematic diagram of heterogeneous screen projection provided by an embodiment of the present application is shown;

[0037] Figure 5 A schematic structural diagram of an electronic device 100 is shown;

[0038] Figure 6 A software structure block diagram of an electronic device provided in an embodiment of the present application is shown;

[0039] Figure 7 A schematic flow chart of a screen projection display method provided in an embodiment of the present application is shown;

[0040] Figure 8 A schematic diagram of a display interface of a security layer under homologous screen projection provided by an embodiment of the present application is shown;

[0041] Fig. 9A schematic diagram of a display interface of a security layer under heterogeneous screen projection provided by an embodiment of the present application is shown;

[0042] Fig.10 A schematic diagram of a chip system provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0044] The terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one or more (including two); "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0045] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0046] The multiple involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first", "second", etc. are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0047] With the development of Internet technology, screen projection technology has been widely used. Among them, screen projection refers to projecting the interface content displayed on the first electronic device to the second electronic device, so that the second electronic device can also synchronously display the interface content displayed by the first electronic device. Through screen projection technology, the interface content of devices with smaller display screens (such as mobile phones and tablets) can be projected onto large-screen display devices (such as TVs and car-mounted multimedia displays), thereby achieving better viewing effects and facilitating users to operate using the operating devices of large-screen devices.

[0048] However, when the interface displayed on the first electronic device includes a security layer, when the interface on the first electronic device is projected to the second electronic device, the display area of ​​the security layer corresponding to the projection interface displayed on the second electronic device is black (blacked out). Therefore, the user cannot recognize the display content of the display area on the second electronic device.

[0049] It should be understood that an interface is composed of multiple layers.

[0050] Below, taking a screen projection initiating device, also known as the source device, and a screen projection device, also known as the target device, as an example, the process of displaying screen projection interface information in the relevant technology is explained. Figure 1 A schematic flow chart of an electronic device displaying interface information in a screen projection scenario in the related art is shown. Figure 1 As shown, assuming that the first electronic device is the source device and the second electronic device is the target device, that is, the first electronic device can simultaneously project the display content in its display interface to the second electronic device for display. In addition to completing its own interface display, the first electronic device also needs to synthesize the display content displayed on the second electronic device.

[0051] Specifically, when the screen projection starts, the interface synthesis system (Surfaceflinger) in the first electronic device will create a virtual screen, and the interface information carried on the virtual screen can be sent to the second electronic device through the network. After obtaining the layer to be synthesized corresponding to the display interface of the first electronic device, Surfaceflinger synthesizes the layer, and then sends the synthesized layer to the hardware graphics renderer (hardwarecomposer, HWC) in the first electronic device. HWC then synthesizes, optimizes and renders the remaining layers, and sends the synthesized, optimized and rendered layer data to the LCD, so that the first electronic device can display the interface corresponding to the synthesized layer.

[0052] In addition, the Surfaceflinger in the first electronic device also needs to send the layers to be synthesized corresponding to the interface information displayed on the virtual screen to the graphics processing unit (GPU) in the first electronic device for synthesizing the remaining layers.

[0053] After the GPU in the first electronic device completes the synthesis of the layers, it will output the data to be encoded, for example, the format of the data to be encoded can be RGBA8888, and send the data to be encoded to the encoder, which encodes and compresses it, and then packages the encoded and compressed data according to the real-time transport protocol (RTP) or the RTP control protocol (RTCP), and then sends the packaged data to the second electronic device through the transmission control protocol (TCP) or the user datagram protocol (UDP). For example, the packaged data can be sent to the second electronic device by wireless transmission.

[0054] After receiving the packaged data sent by the first electronic device, the second electronic device will unpack the received data according to RTP or RTCP, and send the unpacked data to the decoder for decoding, so as to obtain the decoded content, that is, the interface information. The second electronic device sends the obtained interface information to the display system to display the interface information through the LCD.

[0055] Next, when the layers to be synthesized include security layers, the specific process of how Surfaceflinger synthesizes the layers to be synthesized is described.

[0056] Exemplarily, the graphics rendering framework of the first electronic device may be a graphics pipeline rendering framework of an Android system. Of course, the software system may also include but is not limited to Symbian. apple blackberries Hongmeng When the graphics rendering framework of the first electronic device is the graphics pipeline rendering framework of the Android system, Surfaceflinger passes Figure 2 The synthesis process shown in the figure is used to synthesize the layers to be synthesized. Figure 2 Provide specific instructions.

[0057] It is understandable that each window of an Android application corresponds to a canvas, namely a Surface, which can be understood as an interface for the producer and user of graphic data to exchange buffers. Each Surface contains multiple buffers, and the graphics are transmitted through the Buffer as a carrier. Surface is a further encapsulation of the Buffer.

[0058] like Figure 2 As shown in the figure, when a user starts an application to cast the screen, the application in the electronic device will first apply for the handle of the native buffer (Surface) to determine the graphics to be displayed, such as size, position and display content. The Android system generates multiple application graphics data in the interface to be displayed according to the application. One Surface corresponds to one application graphics data, and each application graphics data corresponds to a cache queue (Buffer Queue).

[0059] It should be understood that each Surface includes application graphics data, and the Surface and the application graphics data constitute a layer.

[0060] Exemplarily, the display interface includes 3 application graphics data, or the display interface includes 3 Surfaces, each Surface corresponds to a layer, and each layer corresponds to a Buffer Queue, that is, Surface1→application graphics data 1→Buffer Queue 1→layer 1; Surface2→application graphics data 2→Buffer Queue 2→layer 2; Surface3→application graphics data 3→Buffer Queue 3→layer 3. Among them, → means corresponding to.

[0061] It should be noted that Buffer Queue is equivalent to a glue between Android graphics components, which can be understood as a cache queue. It adopts the producer-consumer model and can adjust the fixed cycle of the buffer from the production queue to the consumption queue. Once the producer transfers its buffer, Surfaceflinger will be responsible for synthesizing all the content into the display part. Among them, Buffer Queue can also be understood as a combination of multiple Buffers.

[0062] Then, Surfaceflinger synthesizes the Buffer Queue generated by rendering according to the window information provided by the window manager service (WMS), that is, synthesizes all layers (corresponding to Surface). It should be understood that Buffer Queue is the link between Surface and layer.

[0063] In the related art, when the layers to be synthesized include a security layer, Surfaceflinger will clear the Buffer Queue data corresponding to the security layer before synthesizing.

[0064] For example, when layer 1 corresponding to Surface1 is a security layer, Surfaceflinger clears the Buffer Queue 1 data in Surface1 before compositing layers 1, 2, and 3. This means that the display information in layer 1 is empty, and the data of the other layers (layers 2 and 3) are retained normally.

[0065] Finally, after the first electronic device completes the synthesis of the layers, it is sent to the second electronic device. After the HWC on the second electronic device processes some layers and performs rendering optimization, the HWC sends the final rendered and optimized layers to the display system of the second electronic device for display. It can be understood that during the layer synthesis process, the display data corresponding to the security layer is cleared, and the data corresponding to the security layer on the display interface is empty. Therefore, on the second electronic device, the display area corresponding to the security layer viewed by the user is a black screen.

[0066] It should be understood that according to different screen projection requirements, the screen projection methods may include same-source screen projection and heterogeneous screen projection, and the above-mentioned synthesis method of the layers to be synthesized including the security layer can be applied to same-source screen projection and heterogeneous screen projection.

[0067] Among them, same-source screen projection refers to projecting one or more application interfaces launched on the first electronic device to the second electronic device. In the same-source screen projection mode, the first electronic device uses one-way encoding to send a standard video stream encoded by the Surface corresponding to all application interfaces rendered on the main screen to the second electronic device, so as to display all application interfaces rendered on the main screen on the display screen of the second electronic device, that is, same-source screen projection is to project the default interface of the first electronic device to the display of the second electronic device, such as mirror projection.

[0068] Figure 3 The schematic diagram of the same-source screen projection provided by the embodiment of the present application is shown. Figure 3Figure (a) in FIG. 3 shows the current output interface 301 of the smart phone, and the interface content 301 is the homepage interface of the smart phone. In response to the user clicking the Taobao application icon (such as Figure 3 After the operation of the "Taobao" icon shown in (a) of the figure, Figure 3 As shown in Figure (b), the smartphone renders the home page interface 302 and the Taobao login interface 3021 included on the home page interface 302 together on the main screen. Based on the same-source screen projection, after the large-screen device receives the standard video stream corresponding to the Surface encoding corresponding to the application interface rendered on the main screen of the smartphone, the large-screen device displays the home page interface 303 on the main screen and the Taobao login interface 3031 included on the home page interface 303 according to the standard video stream. Figure 3 As shown in Figure (c), since the layer corresponding to the Taobao login interface 3031 is a security layer, after the Taobao login interface 3021 on the smart phone is displayed on the large-screen device, the Taobao login interface 3021 is displayed in the form of interface 3031, that is, a black screen interface. The display process containing the security layer can be specifically referred to in Figure 2 The description will not be repeated here.

[0069] It should be understood that Figure 3 In the scenario shown, the Taobao login interface 3021 on the smart phone is displayed in the form of a small window. In another scenario, the Taobao login interface 3021 can also be displayed in full screen. Then, when the Taobao login interface 3021 is displayed in full screen and projected to a large-screen device, the Taobao login interface 3031 of the large-screen device can also be displayed in full screen and be a black screen interface.

[0070] In the heterogeneous screen projection mode, the first electronic device uses two-way encoding, one of which sends the display interface on the display screen of the first electronic device for display, and the other encoding sends the standard video stream and other information corresponding to the application interface rendered on the virtual screen to the second electronic device.

[0071] Figure 4 The following is a schematic diagram of heterogeneous screen projection provided by an embodiment of the present application. Figure 4 FIG. 4 shows the current output interface 401 of the smartphone, where the interface content 401 is the homepage interface of the smartphone. Figure 4 After the operation of the "Huawei Video" icon shown in (a) of FIG. 1 ), the mobile phone renders the main interface 401 on the main screen and the Huawei Video interface (such as Figure 4 The 402 interface and the 4021 interface included in the 402 interface are rendered on the virtual screen. The interface currently visible to the user is Figure 4 The interface shown in Figure (a) is as follows: Figure 4The interface shown in Figure (b) is the interface displayed on the virtual screen, that is, the user-invisible interface. Figure 4 As shown in Figure (c), after the large-screen device (i.e., the second electronic device) receives the standard video stream corresponding to the application interface rendered on the virtual screen from the smart phone (i.e., the first electronic device), the large-screen device displays the application interface on the virtual screen of the first electronic device according to the standard video stream. It should be understood that since the video display interface on the virtual screen includes the security layer 4021, the interface 4031 displayed on the main interface 403 of the large-screen device is a black screen interface. Similarly, the display process of the interface containing the security layer can be specifically referred to Figure 2 The description will not be repeated here.

[0072] It should be understood that Figure 4 In the scenario shown, the Huawei video interface 4021 on the smartphone is displayed in the form of a small window. In another scenario, the Huawei video interface 4021 can also be displayed in full screen. Then, when the Huawei video interface 4021 is displayed in full screen and is projected to a large-screen device, the Huawei video interface 4031 of the large-screen device can also be displayed in full screen and is a black screen interface.

[0073] It can be understood that the same-source screen projection method and the heterogeneous screen projection method each have their own advantages and disadvantages. For example, the same-source screen projection method can ensure the continuity of the application; while the heterogeneous screen projection method requires restarting the application when switching between different screens. For example, application A is displayed on the smartphone, application B is created on the virtual screen, and application B is displayed after projecting to the large-screen device. In this case, application A needs to be restarted on the large-screen device to view application A on the large-screen device. However, the heterogeneous screen projection method has better isolation. For example, the heterogeneous screen projection method can provide users with independent control screens (i.e., the display screen of the first electronic device and the display screen of the second electronic device) to handle different interfaces.

[0074] To sum up, when the layers to be synthesized on the first electronic device include a security layer, after the synthesized layers are sent to the second electronic device and displayed, the display area corresponding to the security layer is in a black screen state, causing the user to be unable to normally view the application content corresponding to the security layer, and even more unable to use the operating device on the second electronic device to operate the display area corresponding to the security layer, which greatly reduces the user experience.

[0075] In view of this, the present application provides a screen projection display method, by searching for a security layer in the layer to be synthesized on the first electronic device, and changing the first value on the security layer flag to a second value, wherein the first value is used to identify or indicate that the layer is a security layer, and the second value is used to indicate that when synthesizing the layer to be synthesized, the security layer is synthesized normally, so that the synthesized interface can normally display the display area corresponding to the security layer. Then, the first electronic device synthesizes the layer to be synthesized based on the second value, and when the first electronic device projects the synthesized layer containing the security layer to the second electronic device, the display area corresponding to the security layer can be displayed normally on the second electronic device. The method provided in the present application allows the user to obtain the display information of the security layer on the second electronic device, and can also operate on the interface where the security layer is located through the hardware of the second electronic device (such as keyboard, mouse, microphone, speaker, etc.), thereby improving the user experience.

[0076] Exemplarily, the present application can use the WMS in the first electronic device to search for a security layer from the layers to be synthesized, and change the first value corresponding to the security layer to a second value. Then, the Surfaceflinger in the first device uses the indication of the second value to synthesize the security layer normally and send it to the second electronic device for display.

[0077] The screen projection display method provided in the embodiment of the present application can be applied to a communication system composed of a first electronic device and a second electronic device. For example, when the first electronic device projects the screen to the second electronic device, the first electronic device and the second electronic device can be connected wirelessly, for example: the first electronic device and the second electronic device can establish a wireless communication connection by "touch", "scan" (such as scanning a QR code or a barcode), "approach automatic discovery" (such as with the help of Bluetooth or wireless fidelity (WiFi)), etc. Among them, the first electronic device and the second electronic device can follow the wireless transmission protocol and transmit information through a wireless connection transceiver. Among them, the wireless transmission protocol may include but is not limited to the Bluetooth (BT) transmission protocol or the wireless fidelity (WiFi) transmission protocol. For example, the WiFi transmission protocol may be a WiFi P2P transmission protocol. The wireless connection transceiver includes but is not limited to Bluetooth, WiFi and other transceivers. Through wireless pairing, information transmission between the first electronic device and the second electronic device is realized. Among them, the information transmitted between the first electronic device and the second electronic device includes but is not limited to content data (such as standard video streams) and control instructions that need to be displayed.

[0078] The first electronic device and the second electronic device include wireless local area networks (WLAN), such as wireless fidelity (Wi-Fi) network, Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR) and other wireless communication solutions.

[0079] Wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), LTE, BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation systems (SBAS).

[0080] It should be understood that the secure screen projection in the embodiments of the present application refers to the screen projection between the first electronic device and the second electronic device through a wired connection. Exemplarily, when both ends of the first electronic device and the second electronic device support the HDCP protocol, it means that the screen projection between the first electronic device and the second electronic device is secure.

[0081] Alternatively, a wired communication connection is established between the first electronic device and the second electronic device via a video graphics array (VGA), a digital visual interface (DVI), a high definition multimedia interface (HDMI), or a data transmission line.

[0082] The wireless connection between the first electronic device and the second electronic device can be understood as when the first electronic device projects the screen to the second electronic device, the screen projection between the first electronic device and the second electronic device is non-secure.

[0083] In an embodiment of the present application, the first electronic device and the second electronic device both include a display screen. The first electronic device and the second electronic device may include, but are not limited to, a smart phone, a netbook, a tablet computer, a smart watch, a smart bracelet, a phone watch, a smart camera, a PDA, a personal computer (personal computer, PC), a personal digital assistant (personaldigital assistant, 0PDA), a portable multimedia player (portable multimedia player, PMP), (augmented reality, AR) / virtual reality (virtual reality, VR) equipment, a television, a projection device, or a somatosensory game console in a human-computer interaction scene, etc. Alternatively, the first electronic device and the second electronic device may also be electronic devices of other types or structures, which are not limited in the present application. Generally, in order to maximize the advantages of screen projection technology, the screen projection display method can be applied between a portable device (i.e., the first electronic device) and a large-screen device (i.e., the second electronic device). For example, the portable device is a smart phone and the large-screen device is a laptop. For another example, the portable device is a tablet computer and the large-screen device is a television. Of course, the present application does not limit the specific devices applicable to the screen projection display method. As mentioned above, the first electronic device and the second electronic device can be any electronic device that supports screen projection, such as a smart phone, a netbook, a tablet computer, a smart watch, a smart bracelet, a phone watch, a smart camera, a PDA, a PMP, an AR / VR device or a television.

[0084] For example, Figure 5The structure diagram of the electronic device 100 (for example, the first electronic device or the second electronic device) is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0085] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0086] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0087] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0088] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0089] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0090] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0091] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0092] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0093] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0094] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0095] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0096] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other first electronic devices, such as AR devices, etc.

[0097] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0098] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the first electronic device through the power management module 141.

[0099] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0100] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0101] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0102] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0103] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0104] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0105] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TDSCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0106] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0107] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0108] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0109] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0110] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0111] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0112] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0113] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0114] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0115] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0116] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0117] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0118] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0119] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0120] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0121] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0122] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0123] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0124] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0125] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0126] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the first electronic device, and is applied to applications such as horizontal and vertical screen switching, pedometers, etc.

[0127] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0128] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0129] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0130] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0131] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0132] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0133] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0134] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0135] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0136] The indicator 192 can be an indicator light, which can be used to indicate the charging status, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0137] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0138] Figure 6 A software structure block diagram of a first electronic device provided in an embodiment of the present application is shown.

[0139] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: application layer, application framework layer, native layer, hardware abstraction layer, and kernel layer (Hardware).

[0140] The application layer can include a series of application packages.

[0141] like Figure 6 As shown, applications include system applications and third-party applications. Applications on electronic devices can be system applications (such as applications installed in electronic devices when the operating system is installed before the electronic device leaves the factory) or third-party applications (such as applications downloaded and installed by users through application stores), which are not limited in the embodiments of this application. For example, applications can include: camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, game, shopping, travel, instant messaging (such as short messages) and other system applications ( Figure 6 The application may also include third-party applications such as Douyin, WeChat, and Taobao ( Figure 6 not shown).

[0142] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. Figure 6As shown, the application framework layer may include a window manager service (WMS), an activity manager service (AMS) and a screen projection management module. In some embodiments, the application framework layer may also include a content provider, a view system, a phone manager, a resource manager, a notification manager, etc. ( Figure 6 not shown).

[0143] Among them, WMS carries window-related data and attributes, and is used to manage window-related states, such as managing window attributes and event dispatching. Among them, managing window programs means outputting to the physical screen or other display devices in an orderly manner with the assistance of the application server and WMS according to the display request of the application. Event dispatching means dispatching user events from the keyboard, physical buttons, touch screen, mouse, trackball (TraceBoll), etc. to the corresponding controls or windows. The window management service can also obtain the display size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0144] Exemplarily, in an embodiment of the present application, the WMS may traverse the flag bits corresponding to the layers to be synthesized, find the security layer by searching for the first value of the security flag bit, and then change the first value of the security layer to the second value.

[0145] For example, the first value may be 0×80, which is used to identify the layer as a security layer. Then, the first value 0×80 of the security layer is changed to a second value 0×30. The second value 0×30 indicates that the security layer is synthesized normally without blackening.

[0146] It should be understood that the first value and / or the second value may also be other identifiers, and the embodiments of the present application are not limited to this.

[0147] AMS is responsible for managing Activities, starting, switching, and scheduling various components in the system, and managing and scheduling applications. Specifically, AMS defines data classes for saving processes, activities, and tasks. Among them, the data class corresponding to the process can include process file information, process memory status information, and the Activities and Services contained in the process. Activity information can be saved in ActivityStack. Among them, Activity Stack is used to uniformly schedule application Activities. ActivityStack can specifically save all running Activities (i.e., final ArrayList mHistory) information, such as interface configuration information. For example, the running Activity can be saved in the new Array List. ActivityStack can also save the information of historically run Activities, such as interface configuration information. It should be noted that Activity does not correspond to an application, but ActivityThread corresponds to an application. Therefore, Android allows multiple applications to run at the same time, which actually allows multiple ActivityThreads to run at the same time.

[0148] In Android, the basic idea of ​​Activity scheduling is this: each application process reports to AMS when starting a new Activity or stopping the current Activity. AMS keeps records for all application processes internally. When AMS receives a start or stop report, it first updates the internal record and then notifies the corresponding client process to run or stop the specified Activity. Since AMS has records of all Activities internally, it can schedule these Activities and automatically close background Activities based on the status of the Activities and system memory.

[0149] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0150] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0151] The phone manager is used to provide communication functions of the first electronic device, such as management of call status (including connecting, hanging up, etc.).

[0152] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0153] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in the status bar, a prompt sound is emitted, the first electronic device vibrates, an indicator light flashes, etc.

[0154] The screen projection management module is responsible for managing matters related to screen projection. For example, the video stream corresponding to the transmission application interface, interface configuration parameters, etc. Exemplarily, the screen projection management module can be Huawei's Assistant or Manager, etc. For example, the Assistant can be a module used to interact with other electronic devices (such as a second electronic device) to exchange information related to screen projection. For example, the Assistant can provide an API and programming framework for the first electronic device to communicate with other electronic devices (such as a second electronic device). Exemplarily, when the screen projection device is a computer, the Manager can be a computer manager, a computer assistant, etc. In some possible embodiments, the screen projection management module can not only project programs running on the local device to other devices, but also receive and parse video streams and related information sent by other devices, thereby displaying applications running on other devices on the local device. In some possible embodiments, the screen projection management module is cut and can only implement one of the functions in the above embodiments, so as to meet the needs of different device forms for screen projection functions.

[0155] The native layer includes the interface service (Surfaceflinger) and the hardware renderer (hardware composer, HWC). Each application may correspond to one or more graphical interfaces, and each interface is called a surface. In the Android implementation, Surfaceflinger is a service that provides system-wide interface composition (surfacecomposer) functions, which can combine 2D and 3D surfaces of various applications. Surfaceflinger can delegate some synthesis work to HWC to share the workload on OpenGL and GPU.

[0156] In an embodiment of the present application, Surfaceflinger synthesizes the layer to be synthesized corresponding to the display interface. When Surfaceflinger traverses the flag bits of each layer, the BufferQueue data of the security layer corresponding to the flag bit 0×30 is synthesized normally, so that the display area corresponding to the security layer can be viewed normally from the user's perspective.

[0157] The hardware abstraction layer (HAL) is used to abstract the hardware. For example, the hardware abstraction layer can include the audio hardware abstraction layer, Bluetooth hardware abstraction layer, camera hardware abstraction layer, hardware renderer abstraction layer, and other hardware device abstraction layers; the hardware renderer abstraction layer can be connected to the hardware renderer algorithm library. When the graphic data is displayed on the device screen, the hardware renderer abstraction layer can call the algorithm in the hardware renderer algorithm library for rendering. The hardware renderer is the core of all Android graphics rendering.

[0158] The kernel layer is the layer between hardware and software. The kernel layer can include display drivers, input / output device drivers (for example, keyboards, touch screens, headphones, speakers, microphones, etc.), device nodes, camera drivers, audio drivers, and sensor drivers. Users perform input operations through input devices, and the kernel layer can generate corresponding raw input events based on the input operations and store them in device nodes.

[0159] It should be noted that Figure 6 Taking the layered Android system as an example, this paper introduces the application interface display process in a projection display scenario. This application does not limit the specific architecture of the electronic device software system. For the specific introduction of software systems with other architectures, you can refer to conventional technologies. For example, the software system may also include but is not limited to Symbian apple blackberries Hongmeng For ease of understanding, the following embodiments of this application will be based on an operating system with Figure 5 and Figure 6 Taking the electronic device with the shown structure as an example, the screen projection display method provided in the embodiment of the present application is specifically explained in combination with the accompanying drawings and application scenarios.

[0160] Figure 7 A schematic flow chart of a screen projection display method provided in an embodiment of the present application is shown. Figure 7 As shown, the method 700 includes S710 - S760 .

[0161] S710: The first electronic device and the second electronic device establish a screen projection connection.

[0162] In the embodiment of the present application, the first electronic device and the second electronic device may be connected in a wired manner or in a wireless manner. The connection manner between the first electronic device and the second electronic device will not be described in detail here.

[0163] S720, the first electronic device determines a security layer in the layers to be synthesized of the screen projection interface, and the flag bit of the security layer is a first value.

[0164] It should be understood that the first electronic device projects the screen onto the second electronic device by projecting the interface to be displayed created by the first electronic device onto the second electronic device. It should also be understood that the interface to be displayed includes multiple layers to be synthesized.

[0165] When the user performs screen projection, the first electronic device generates multiple layers to be synthesized corresponding to the interface to be displayed in response to the user's operation of starting the application. Before the first electronic device projects the interface to be displayed to the second electronic device, the first electronic device traverses the layers to be synthesized and finds the security layer from the layers to be synthesized.

[0166] Optionally, in a possible implementation, the first electronic device may determine whether the layers to be synthesized include a security layer through a flag bit corresponding to each layer to be synthesized.

[0167] It should be understood that according to the native requirements of Android, when the layers to be synthesized include a security layer, a security declaration needs to be made for the security layer. Generally, the security layer will be marked with a security flag.

[0168] Exemplarily, the first value of the security flag may be 0×80. Of course, the security flag may also be marked by other identifiers, or the first value may also be other values. This embodiment of the application does not limit this.

[0169] It should also be understood that the first value of the security flag of the security layer is used to instruct Surfaceflinger to clear the Buffer Queue data of the security layer when synthesizing the layer to be synthesized, so that after synthesizing the layer to be synthesized, there is no data for display information in the security layer, and finally the display area corresponding to the security layer projected on the second electronic device is black. In addition, the first value of the security flag is also used to indicate that the display area corresponding to the security layer cannot be screenshotted, cannot be recorded, and cannot be displayed in multiple tasks.

[0170] In one possible implementation, when a user starts an application, the AMS in the first electronic device calls the start activity interface to start the activity, and then the WMS in the first electronic device traverses the layers to be synthesized in the application to determine whether the layers to be synthesized include a security layer.

[0171] Exemplarily, the WMS in the first electronic device may traverse the flag bit of each layer to be synthesized in the application program, and determine whether the layer to be synthesized includes the security flag bit of the security layer.

[0172] S730: The first electronic device changes the first value to a second value.

[0173] In an embodiment of the present application, after the first electronic device finds the security layer, the security flag corresponding to the security layer can be changed so that the security layer can be normally synthesized on the first electronic device. Finally, after the display interface is projected to the second electronic device, the display area corresponding to the security layer can be normally displayed on the second electronic device.

[0174] In some possible implementations, the first electronic device may change the first value corresponding to the security layer to a second value.

[0175] It should be understood that the first value is used to indicate that the layer is a security layer. Then, the first electronic device performs security processing when synthesizing the security layer, that is, clears the information of the display area corresponding to the security layer. In addition, the first electronic device ensures that the security layer cannot be screenshot, cannot be recorded, and cannot be displayed in multiple tasks. The second value is used to indicate that when the first electronic device synthesizes the layer to be synthesized, the display data corresponding to the security layer in the layer to be synthesized is displayed normally.

[0176] It should also be understood that the second value is only used as an indicator to instruct the first electronic device to display the display area information corresponding to the security layer normally when synthesizing the layer to be synthesized, and to ensure that the security layer cannot take screenshots, cannot record screens, and cannot display multiple tasks. That is, regardless of whether the value of the security flag corresponding to the security layer is the first value or the second value, the security layer cannot take screenshots, cannot record screens, and cannot display multiple tasks.

[0177] As a possible implementation, the first value is used to instruct the Surfaceflinger in the first electronic device to clear the Buffer Queue data in the security layer when synthesizing the layer to be synthesized, so that after the layer to be synthesized is synthesized, there is no data for display information in the security layer, and finally the display area corresponding to the security layer placed on the second electronic device is black. The second value is used to instruct the Surfaceflinger in the first electronic device to display the Buffer Queue data in the security layer normally when synthesizing the layer to be synthesized, so that after the interface to be displayed is synthesized, the security layer retains the data for display information, and finally the display area corresponding to the security layer placed on the second electronic device is displayed normally.

[0178] For example, the first value may be the security flag bit 0×80 of the security layer in step S720, and the second value may be 0×30. Of course, the second value may also be other values, which are not limited in the embodiment of the present application.

[0179] In this embodiment, the security layer in the layer to be synthesized is found by searching the flags of all the layers to be synthesized of the interface to be displayed in the first electronic device. Then, the first value of the security flag corresponding to the security layer is changed to the second value, so that the first electronic device can synthesize the security layer normally when synthesizing all the layers to be synthesized, and finally after the screen is projected on the second electronic device, the second electronic device can display the area corresponding to the security layer normally. This implementation method of changing the value of the security flag can only change the synthesis method of the security layer without changing the other properties of the security layer, for example, the display area corresponding to the security layer cannot be screenshot, cannot record the screen, and cannot display multiple tasks. Thereby ensuring that the security layer can be displayed normally and the security of the security layer is guaranteed.

[0180] S740: The first electronic device synthesizes the layer to be synthesized according to the second value, so that the display area corresponding to the security layer is displayed normally.

[0181] When the first electronic device detects that the security flag corresponding to the security layer in the layer to be synthesized is a second value, the first electronic device synthesizes the security layer on the layer to be synthesized normally when synthesizing the layer to be synthesized, that is, retains the display information of the security layer, so that the display area corresponding to the security layer on the layer to be synthesized projected onto the second electronic device can be displayed normally.

[0182] As a possible implementation method, when synthesizing the layers to be synthesized, the Surfaceflinger on the first electronic device traverses the layers to be synthesized, and when the security flag of the traversed security layer is the second value, the security layer on the layers to be synthesized is synthesized normally.

[0183] Specifically, when the Surfaceflinger on the first electronic device synthesizes the layers to be synthesized, the Surfaceflinger first traverses the layers to be synthesized. When it is found that the security flag bit of the traversed security layer is the second value, the Surfaceflinger will synthesize all layers normally according to the Buffer Queue data corresponding to the security layer, instead of Figure 2As shown, when Surfaceflinger synthesizes the layer to be synthesized, the Buffer Queue data corresponding to the security layer is cleared. It should be understood that after Surfaceflinger synthesizes the security layer normally according to the instruction of the second value, the second electronic device can normally display the display area corresponding to the security layer after receiving the synthesized layer from the first electronic device through local rendering.

[0184] It should be noted that Surfaceflinger can synthesize all layers, and Surfaceflinger can also delegate some synthesis work to HWC for layer synthesis. For example, Surfaceflinger can provide a complete list of all layers to HWC, allowing HWC to decide which layers to synthesize based on its hardware capabilities.

[0185] S750, the first electronic device sends screen projection data to the second electronic device, where the screen projection data includes the synthesized layers.

[0186] The first electronic device sends the projection data to the second electronic device, so that the second electronic device displays the projection interface corresponding to the interface to be displayed. In an embodiment of the present application, the projection data includes a synthesized layer.

[0187] In one possible implementation, after the Surfaceflinger in the first electronic device synthesizes the layer to be synthesized, the synthesized layer can be sent to the HWC of the second electronic device. After rendering by the HWC of the second electronic device, the second electronic device displays the projection interface corresponding to the interface to be displayed.

[0188] Optionally, when sending the synthesized layer, the first electronic device can encode the synthesized layer into a video stream, and package the encoded and compressed video stream data according to the real-time transport protocol RTP or RTCP control protocol, and then send the packaged video stream to the second electronic device.

[0189] Specifically, after the Surfaceflinger in the first electronic device synthesizes all the layers to be synthesized, the synthesized layers are sent to the encoder to generate a standard video stream. Then, the first electronic device performs RTP or RTCP packaging on the obtained standard video stream. Finally, the first electronic device sends the packaged data to the second electronic device via TCP or UDP.

[0190] In this embodiment, since the first electronic device encodes and compresses the synthesized layer and performs RTP or RTCP packaging processing on the encoded layer, and then sends the packaged layer to the second electronic device, it can not only improve the security of the layer data, but also reduce the amount of data transmission and improve the efficiency of data transmission.

[0191] S760, the second electronic device displays the screen projection interface.

[0192] In the embodiment of the present application, after receiving the video stream of the synthesized layers sent by the first electronic device, the second electronic device needs to parse the video stream to obtain all the layers that need to be displayed.

[0193] Exemplarily, the second electronic device may unpack the video stream of the synthesized layers according to RTP or RTCP, and decode the unpacked layers to obtain display information of all layers.

[0194] In a possible implementation, if the first electronic device uses RTP to package the encoded layer, the second electronic device will use RTP to unpack the received video stream of the synthesized layer, and if the first electronic device uses RTCP to package the encoded layer, the second electronic device will use RTCP to unpack the received video stream of the synthesized layer. After that, the unpacked layer is sent to the decoder to decode the unpacked layer.

[0195] Finally, the HWC in the second electronic device outputs the decoded video stream data of all layers to the frame buffer, and then outputs it to the second electronic device for display. Alternatively, the HWC directly outputs all the synthesized layer data to the display screen on the second electronic device for display.

[0196] Optionally, in a possible implementation, the first electronic device may further send the layer to be synthesized to the second electronic device, and the second electronic device normally synthesizes the layer to be synthesized corresponding to the interface to be displayed according to the second value.

[0197] Specifically, the first electronic device may encode the layer to be synthesized into a video stream, and package the encoded and compressed video stream data according to the real-time transport protocol RTP or RTCP control protocol, and then send the packaged video stream to the second electronic device.

[0198] After receiving the video stream of the layer to be synthesized sent by the first electronic device, the second electronic device needs to parse the video stream to obtain all the layers to be synthesized, and then synthesize the layers to be synthesized to obtain the synthesized layers.

[0199] Finally, the second electronic device renders the synthesized layer in the local HWC, and directly outputs the rendered layer to the display screen on the second electronic device for display. Alternatively, the HWC outputs the rendered layer to the frame buffer, and then outputs it to the second electronic device for display.

[0200] Optionally, as another possible implementation method, after the Surfaceflinger on the first electronic device synthesizes all the layers, it can also complete the synthesis, optimization and rendering of the remaining layers through the local HWC.

[0201] Afterwards, the HWC on the first electronic device sends the rendered optimized layer to the Surfaceflinger on the first electronic device, which then sends it to the WMS. The WMS sends the synthesized layer to be synthesized to the Figure 4 The screen projection management module of the first electronic device shown is used to send the synthesized layer to be synthesized to the second electronic device (such as the screen projection management module of the second electronic device) for direct display.

[0202] In this implementation method, the first electronic device does not need to encode, compress, and package the synthesized layers and send them to the second electronic device, which then needs to be decoded and decompressed by the second electronic device before rendering and displaying them, thereby improving the efficiency of screen projection display.

[0203] The screen projection display method provided by the embodiment of the present application is a screen projection display method, which searches for a security layer in the layer to be synthesized on the first electronic device, and changes the first value of the security flag corresponding to the security layer to a second value, wherein the first value is used to identify or indicate that the layer is a security layer, and the second value is used to indicate that when synthesizing the layer to be synthesized, the security layer is synthesized normally, so that the synthesized interface can normally display the display area corresponding to the security layer. Then, the first electronic device synthesizes the layer to be synthesized based on the second value, and when the first electronic device projects the synthesized layer containing the security layer to the second electronic device, the display area corresponding to the security layer can be normally displayed on the second electronic device. The method provided by the present application can enable the user to obtain the display information of the security layer on the second electronic device, and can also operate on the interface where the security layer is located through the hardware of the second electronic device (such as keyboard, mouse, microphone, speaker, etc.), thereby improving the user experience.

[0204] Optionally, before the first electronic device executes step S720, the first electronic device may determine whether the screen projection between the first electronic device and the second electronic device is non-secure.

[0205] It should be understood that in the related art, when the first electronic device and the second electronic device are in a secure screen projection, and the layers to be synthesized when the first electronic device projects the screen to the second electronic device include a secure layer, the display area corresponding to the secure layer is normally displayed on the second electronic device. When the first electronic device and the second electronic device are in a non-secure screen projection, and the layers to be synthesized when the first electronic device projects the screen to the second electronic device include a secure layer, the display area corresponding to the secure layer displayed on the second electronic device is black.

[0206] Therefore, in an embodiment of the present application, it can be determined whether the screen projection between the first electronic device and the second electronic device is secure. If the screen projection between the first electronic device and the second electronic device is non-secure, the screen projection display method provided in the embodiment of the present application can be executed.

[0207] In a possible implementation, when the first electronic device and the second electronic device are wirelessly connected, it indicates that the screen projection between the first electronic device and the second electronic device is non-secure.

[0208] Exemplarily, the first electronic device and the second electronic device can determine whether the first electronic device and the second electronic device are wirelessly connected by determining whether the HDCP protocol is supported. When the first electronic device and the second electronic device are wirelessly connected, it indicates that the screen projection between the first electronic device and the second electronic device is non-secure.

[0209] Specifically, when both ends of the first electronic device and the second electronic device support the HDCP protocol, it means that the first electronic device and the second electronic device are connected by wire, which means that the screen projection between the first electronic device and the second electronic device is secure; when at least one end of the first electronic device and the second electronic device does not support the HDCP protocol, it means that the first electronic device and the second electronic device are connected by wireless, which means that the screen projection between the first electronic device and the second electronic device is non-secure.

[0210] In an embodiment of the present application, when a first electronic device projects a screen to a second electronic device, the first electronic device and the second electronic device respectively determine whether their respective ends support the HDCP protocol, and then the second electronic device sends the determination result to the first electronic device. When the first electronic device determines that at least one of the first electronic device and the second electronic device does not support the HDCP protocol, it indicates that the screen projection between the first electronic device and the second electronic device is non-secure.

[0211] When the screen projection between the first electronic device and the second electronic device is secure, the flag bit of the screen projection interface created by the first electronic device is secure, and the flag bit is used to indicate that the screen projection interface is a secure interface, which means that the screen projection between the first electronic device and the second electronic device is secure; when the flag bit of the screen projection interface is not secure, the flag bit is used to indicate that the screen projection interface is a non-secure interface, which means that the screen projection between the first electronic device and the second electronic device is non-secure.

[0212] It should be understood that in the embodiment of the present application, the screen to be projected interface can also be understood as the display to be projected.

[0213] Of course, other methods may also be used to determine whether the screen projection between the first electronic device and the second electronic device is secure, and the embodiments of the present application do not limit this.

[0214] The screen projection display method provided in the present application can be applied to the same-source screen projection or heterogeneous screen projection scenarios. The following takes the first electronic device as a smart phone and the second electronic device as a large-screen device as an example to specifically describe the application scenario of the screen projection display method. Figure 8 The following is a schematic diagram showing the display interface of the security layer under the same-source screen projection provided by the embodiment of the present application. Figure 8 Figure (a) in FIG. 8 shows the current output interface 801 of the smart phone, and the interface content 801 is the homepage interface of the smart phone. In response to the user clicking the Taobao application icon (such as Figure 8 After the operation of the "Taobao" icon shown in (a) of the figure, Figure 8 As shown in Figure (b), the smartphone renders the homepage interface 802 and the Taobao login interface 8021 on the homepage interface 802 together on the main screen. In response to the user's screen projection operation, the WMS on the smartphone detects the flags of all layers on the interface 802. When it is detected that the flag of the security layer 8021 on the interface 802 is 0×80, it indicates that the Taobao login interface 8021 is a security layer, and then the security flag of the security layer is changed to 0×30. The 0×30 indicates that the security layer is synthesized normally during synthesis. Then, when the Surfaceflinger on the smartphone performs multi-layer synthesis, it traverses the flags of all layers, and when it detects that the flag of the layer is 0×30, the layer is synthesized normally. Finally, Surfaceflinger encodes the data corresponding to the synthesized layer into a standard video and sends it to the large-screen device.

[0215] Based on the same-source screen projection, after the large-screen device receives the standard video stream corresponding to the Surface encoding corresponding to the application interface rendered by the main screen of the smart phone, the large-screen device displays the home page interface 803 and the Taobao login interface 8031 ​​included on the home page interface 803 according to the standard video stream. Figure 8 As shown in Figure (c), although the layer corresponding to the Taobao login interface is a security layer, the Taobao login interface 8031 ​​can still be displayed normally after being displayed on a large-screen device, and the security layer cannot take screenshots, cannot record the screen, and cannot display multiple tasks.

[0216] It should be understood that Figure 8 In the scenario shown, the Taobao login interface 8021 on the smart phone is displayed in the form of a small window. In another scenario, the Taobao login interface 8021 can also be displayed in full screen. Then, when the Taobao login interface 8021 is displayed in full screen and is projected to a large-screen device, the Taobao login interface 8031 ​​of the large-screen device can also be displayed in full screen. Although the layer corresponding to the Taobao login interface is a security layer, the Taobao login interface 8031 ​​can still be displayed normally after being displayed on the large-screen device.

[0217] Fig. 9 The following is a schematic diagram of the display interface of the security layer under the heterogeneous screen projection provided by the embodiment of the present application. Fig. 9 Figure (a) in FIG. 1 shows the current output interface 901 of the smartphone, and the interface content 901 is the homepage interface of the smartphone. In response to the user clicking the Huawei video application icon (such as Fig. 9 After the operation of the "Huawei Video" icon shown in (a) of FIG. 1 ), the mobile phone renders the mobile phone 901 homepage on the main screen 902, and the Huawei video interface (such as Fig. 9 Taking the documentary "Brave Step" shown in Figure (b) as an example) 9021 is rendered on the virtual screen. In response to the user's screen projection operation, the WMS on the smartphone detects the layer flag on the interface 902. When it is detected that the flag of the security layer 9021 on the interface 902 is 0×80, it indicates that the video playback interface 9021 is a security layer, and then the security flag of the security layer is changed to 0×30. The 0×30 indicates that the security layer is synthesized normally during synthesis. Then, when the Surfaceflinger on the smartphone performs multi-layer synthesis, it traverses the flags of all layers, and when it detects that the flag of the layer is 0×30, the layer is synthesized normally. Finally, the Surfaceflinger on the smartphone encodes the data corresponding to the synthesized layer into a standard video stream and sends it to the large-screen device.

[0218] Based on heterogeneous screen projection, such as Fig. 9As shown in Figure (c), after the large-screen device (i.e., the second electronic device) receives the standard video stream corresponding to the application interface rendered on the virtual screen from the smart phone (i.e., the first electronic device), the large-screen device displays the video application interface according to the standard video stream. It should be noted that although the video display interface on the virtual screen includes a security layer, the display interface 9031 corresponding to the security layer on the main interface 903 of the large-screen device is displayed normally, and the security layer cannot be screenshot, cannot be recorded, and cannot be displayed in multiple tasks.

[0219] It should be understood that Fig. 9 In the scenario shown, the Huawei video interface 9021 on the smartphone is displayed in the form of a small window. In another scenario, the Huawei video interface 9021 can also be displayed in full screen. Then, when the Huawei video interface 9021 is displayed in full screen and is projected to a large-screen device, the Huawei video interface 9031 of the large-screen device can also be displayed in full screen. Although the layer corresponding to the Huawei video interface is a security layer, the Huawei video interface 9031 can still be displayed normally after being displayed on the large-screen device.

[0220] The present application also provides a chip system, such as Fig.10 As shown, the chip system includes at least one processor 1001 and at least one interface circuit 1002. The processor 1001 and the interface circuit 1002 can be interconnected via lines. For example, the interface circuit 1002 can be used to receive signals from other devices (such as the memory of any of the above-mentioned electronic devices). For another example, the interface circuit 1002 can be used to send signals to other devices (such as the processor 1001). Exemplarily, the interface circuit 1002 can read the instructions stored in the memory and send the instructions to the processor 1001. When the instructions are executed by the processor 1001, the electronic device can execute the various steps performed by any of the electronic devices in the above-mentioned embodiments. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0221] It should also be understood that the division of units in the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And the units in the device can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some units can also be implemented in the form of software calling through processing elements, and some units can be implemented in the form of hardware. For example, each unit can be a separately established processing element, or it can be integrated in a certain chip of the device. In addition, it can also be stored in the memory in the form of a program, and called and executed by a certain processing element of the device. The processing element here can also be called a processor, which can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each unit above can be implemented by an integrated logic circuit of hardware in the processor element or in the form of software calling through a processing element. In one example, the unit in any of the above devices may be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA), or a combination of at least two of these integrated circuit forms. For another example, when the unit in the device can be implemented in the form of a processing element scheduler, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call a program. For another example, these units can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0222] The present application also provides a device, which is included in an electronic device (for example, a first electronic device or a second electronic device), and has the function of implementing the behavior of the electronic device in any of the above embodiments. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes at least one module or unit corresponding to the above function.

[0223] The present application also provides an electronic device (for example, a first electronic device or a second electronic device), which includes the device provided in the above-mentioned embodiment of the present application.

[0224] The embodiment of the present application also provides a computer-readable storage medium for storing computer program code, the computer program including instructions for executing the steps of an electronic device (e.g., a first electronic device or a second electronic device) performing a display interface in any of the embodiments provided in the above embodiments of the present application. The readable medium may be a read-only memory (ROM) or a random access memory (RAM), which is not limited in the embodiment of the present application.

[0225] The present application also provides a computer program product, which includes instructions. When the instructions are executed, the electronic device executes the steps of executing or displaying an interface by the electronic device in any of the above embodiments.

[0226] The embodiment of the present application further provides a chip, which includes: a processing unit and a communication unit, the processing unit, for example, may be a processor, and the communication unit, for example, may be an input / output interface, a pin or a circuit, etc. The processing unit may execute computer instructions to enable the electronic device to execute any of the screen projection display methods provided in the above-mentioned embodiment of the present application.

[0227] Optionally, the computer instructions are stored in a storage unit.

[0228] Optionally, the storage unit is a storage unit within the chip, such as a register, a cache, etc. The storage unit may also be a storage unit within the terminal that is located outside the chip, such as a ROM or other types of static storage devices that can store static information and instructions, random RAM, etc. Among them, the processor mentioned in any of the above may be a CPU, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned electronic device projection screen display method. The processing unit and the storage unit may be decoupled and respectively arranged on different physical devices, and connected by wired or wireless means to implement the respective functions of the processing unit and the storage unit, so as to support the system chip to implement the various functions in the above-mentioned embodiments. Alternatively, the processing unit and the memory may also be coupled on the same device.

[0229] Among them, the electronic device, device, computer-readable storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0230] An embodiment of the present application also provides a graphical user interface on an electronic device, the electronic device having a display screen, a camera, a memory, and one or more processors, the one or more processors being used to execute one or more computer programs stored in the memory, the graphical user interface comprising a graphical user interface displayed when the electronic device executes steps as performed by the electronic device in any of the above embodiments.

[0231] It is understandable that, in order to realize the above functions, the above-mentioned electronic devices, etc. include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0232] The embodiment of the present application can divide the functional modules of the above-mentioned electronic device etc. according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0233] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0234] Each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0235] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk.

[0236] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for screen projection display, characterized in that: Applied to a first electronic device, the first electronic device establishes a screen projection connection with a second electronic device, and the method includes: Determine a security layer in the layers to be synthesized of the screen interface to be projected, wherein the flag bit of the security layer is a first value; wherein, in response to the user's operation of starting the application program, generate a plurality of layers to be synthesized corresponding to the screen interface to be projected; changing the first value to a second value; According to the second value, synthesizing the layer to be synthesized so that the display area corresponding to the security layer is displayed normally; Sending screen projection data to the second electronic device, wherein the screen projection data includes the synthesized layer.

2. The method according to claim 1, characterized in that Before determining the security layer in the layer to be synthesized of the screen interface to be projected, the method further includes: Determine that the screen projection between the first electronic device and the second electronic device is non-secure.

3. The method according to claim 1, characterized in that In the synthesized layer, the display area corresponding to the security layer cannot be screenshotted, cannot be recorded, and cannot be multi-tasking displayed.

4. The method according to any one of claims 1 to 3, characterized in that The window management service WMS in the first electronic device changes the first value to the second value.

5. The method according to any one of claims 1 to 3, characterized in that: The interface synthesis system surfaceflinger in the first electronic device synthesizes the layer to be synthesized according to the second value.

6. The method according to any one of claims 1 to 3, characterized in that The first value includes: 0×80.

7. The method according to any one of claims 1 to 3, characterized in that The second value includes: 0×30.

8. The method according to claim 2 or 3, characterized in that: When the first electronic device and the second electronic device are wirelessly connected, the first electronic device is in a non-secure screen projection state.

9. The method according to claim 2 or 3, characterized in that: The flag bit in the screen to be projected interface is used to indicate that when the screen to be projected interface is a non-secure interface, the first electronic device is in a non-secure screen projection state.

10. The method according to any one of claims 1 to 3, characterized in that: The first electronic device projects the screen to the second electronic device in a same-source screen projection manner or a different-source screen projection manner.

11. A screen projection display system, characterized in that: The system includes a first electronic device and a second electronic device, wherein the first electronic device is used to project a screen onto the second electronic device; The first electronic device is used to determine a security layer in the layers to be synthesized of the screen interface to be projected, wherein the flag bit of the security layer is a first value; wherein, in response to the user's operation of starting the application program, a plurality of layers to be synthesized corresponding to the screen interface to be projected are generated; The first electronic device is further used to change the first value to a second value; The first electronic device is further used to synthesize the layer to be synthesized according to the second value, so that the display area corresponding to the security layer is displayed normally; The first electronic device is further used to send screen projection data to the second electronic device, where the screen projection data includes the synthesized layer.

12. The system according to claim 11, characterized in that Before the first electronic device determines the security layer in the layer to be synthesized of the screen projection interface, the first electronic device is also used to determine that the screen projection between the first electronic device and the second electronic device is non-secure.

13. The system according to claim 11, characterized in that In the synthesized layer, the display area corresponding to the security layer cannot be screenshotted, cannot be recorded, and cannot be multi-tasking displayed.

14. The system according to any one of claims 11 to 13, characterized in that: The WMS in the first electronic device is used to change the first value to the second value.

15. The system according to any one of claims 11 to 13, characterized in that The surfaceflinger in the first electronic device is used to synthesize the layer to be synthesized according to the second value.

16. The system according to any one of claims 11 to 13, characterized in that: The first value includes: 0×80.

17. The system according to any one of claims 11 to 13, characterized in that: The second value includes: 0×30.

18. The system according to claim 12 or 13, characterized in that When the first electronic device and the second electronic device are wirelessly connected, the first electronic device is in a non-secure screen projection state.

19. The system according to any one of claims 12-13, characterized in that: The flag bit in the screen to be projected interface is used to indicate that when the screen to be projected interface is a non-secure interface, the first electronic device is in a non-secure screen projection state.

20. The system according to any one of claims 11 to 13, characterized in that The first electronic device projects the screen to the second electronic device in a same-source screen projection manner or a different-source screen projection manner.

21. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory is used to store instructions, and the processor is used to read the instructions to execute the method according to any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 10.

23. A chip, characterized in that: include: A processor, configured to call and run a computer program from a memory, so that a communication device equipped with the chip executes a method as claimed in any one of claims 1 to 10.

Images