Display method and electronic device

Abstract

Embodiments of the present application disclose a display method and an electronic device. The first device disclosed in the embodiments of the present application can project the screen to the second device. Specifically, after receiving the screen projection instruction input by the user, the first device projects the first window interface to the second device. When the first device receives the instruction to run the second window interface, after determining that the second window interface includes a security flag bit, it can send the task number of the second window interface and a security protection instruction to the second device. Furthermore, the second device displays the first window interface and a security interface, and the security interface is the screen projection interface of the second window interface. In this way, the second device can not only safely display the interface with security requirements, but also ensure the normal operation of the user on the interface without security requirements, improving the user experience.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of displays, and in particular, to a display method and an electronic device. Background Art

[0002] Screen mirroring is a technology in which Device 1 transmits the window interface of Device 1 to be displayed on Device 2. Exemplarily, as Figure 1 shown, a mobile phone can transmit the window interface of the mobile phone to be displayed on a computer. In this way, the computer can display the screen mirroring interface 01 of the mobile phone window interface on the screen end. The user can operate the application (APP) of the mobile phone through the screen mirroring interface 01.

[0003] Among them, the APPs running on the mobile phone side include APPs with security requirements, such as APPs that require users to enter passwords. Based on this, in the scenario of screen mirroring, the passwords entered on the mobile phone side will be displayed on the computer, such as visible passwords / number of password digits / biometric passwords, etc., which brings potential risks of privacy leakage to users. Summary of the Invention

[0004] Embodiments of the present application provide a display method and an electronic device, which can solve the problem that the existing display method leads to poor user experience.

[0005] In the embodiments of the present application, the first device can screen mirror multiple window interfaces of the first device end to the second device.

[0006] In a first aspect, embodiments of the present application provide a display method, the method includes: the first device receives a screen mirroring instruction input by a user; in response to the screen mirroring instruction, the first device screens the first window interface to the second device; the first device runs a second window interface in response to an instruction; the first device determines that the second window interface includes a security flag bit; the first device sends a task number corresponding to the second window interface and a security protection instruction to the second device, and the security protection instruction is used to instruct the second device to display the screen mirroring interface of the second window interface as a secure interface.

[0007] Among them, the first device can configure an application display area (display), and configure a display policy of the window interface moving at the first device end within the display. Furthermore, the first device can screen mirror to the second device based on the display. In addition, the first device configures a task ID for the APP moving at the first device end to represent the window interfaces of each APP.

[0008] In the embodiment of the present application, before screen mirroring, the first device determines whether each window interface has security requirements. Further, based on the detection result, the first device sends the task ID of each window interface and the screen mirroring information of the window interface to the second device. The second device displays a security interface and a screen mirroring interface containing the content of the window interface based on the task ID, where the security interface is the screen mirroring interface of the window interface with security requirements. In this way, the second device can not only securely display the interface with security requirements but also ensure that the user can normally operate the interface without security requirements.

[0009] In a possible design, the first device runs the second window interface in response to an instruction, including: the first device receives a first operation instruction from the second device, the first operation instruction is associated with a first application APP, and the second window interface is a window interface of the first APP; the first device runs the first APP to start the second window interface. Among them, the first device can communicate with the second device based on the WiFi Direct technology. In this way, it can be realized that the user operates the APP of the first device at the second device end.

[0010] In a possible design, the first device runs the second window interface in response to an instruction, including: the first device receives a second operation instruction input by the user, the second operation instruction corresponds to a first application APP, and the second window interface is a window interface of the first APP; the first device runs the first APP and displays the second window interface.

[0011] In a possible design, after the first device sends the task number corresponding to the second window interface and a security protection instruction to the second device, it further includes: the first device receives a third operation instruction from the second device, the third operation instruction contains the task number corresponding to the second window interface; in response to the third operation instruction, the first device displays the second window interface; the first device receives the operation information input by the user based on the second window interface; the first device updates the second window interface to a third window interface; the first device screen-mirrors the third window interface to the second device. After the second device displays the screen mirroring interface of the second window interface as a security interface, the user can trigger the second window interface to be displayed on the display screen of the first device through the second device to continue operating the second window at the first device end. And the first device can determine the second window interface through the task number. By adopting this implementation method, the first device associates the window interface with the task ID, so that both the first device and the second device can determine the window interface through the task ID, which facilitates the user to switch the window interfaces displayed at the first device end and the second device end, thereby improving the user experience.

[0012] In a possible design, the first device determines that the second window interface includes a security flag bit, including: the first device calls the window management service of the first device to traverse the window states of the second window interface; if the window state of the second window interface includes a security flag, the first device determines that the second window interface includes a security flag bit.

[0013] In a possible design, the security flag includes a first flag, a second flag, and a third flag. The first flag indicates an application security lock, the second flag indicates an application active setting, and the third flag indicates a call to a security keyboard.

[0014] In a second aspect, an embodiment of the present application provides a display method, and the method includes: a second device receives a video stream of a first window interface from a first device; the second device displays an image corresponding to the video stream to obtain a screen mirroring interface of the first window interface; the second device receives a task number and a security protection instruction from the first device; the second device displays the screen mirroring interface and a security interface of the first window interface.

[0015] In a possible design, after the second device displays the screen mirroring interface and the security interface of the first window interface, it further includes: the second device receives a click instruction input by a user, and the click instruction corresponds to the security interface; the second device generates an operation instruction, and the operation instruction includes a task number; the second device sends the operation instruction to the first device.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, and the electronic device has the functions of the first device in the above method. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the structure of the above electronic device includes a processor and a transceiver. The processor is configured to process the functions corresponding to the first device in the above method executed by the electronic device. The transceiver is used to implement information interaction with the second device. The electronic device may further include a memory, and the memory is used to be coupled with the processor and store necessary program instructions and data of the electronic device.

[0017] Fourth aspect, embodiments of the present application provide an electronic device, which has the functions of the second device in the above method. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In a possible design, the structure of the above electronic device includes a processor, a transceiver, and a display. The processor is configured to process the corresponding functions of the electronic device executing the above method. The transceiver is used to implement information interaction with the first device. The display is used to implement the display of the screen mirroring interface and the security interface. The electronic device may further include a memory, which is used to be coupled with the processor and stores necessary program instructions and data of the electronic device.

[0018] Fifth aspect, embodiments of the present application provide a computer storage medium, in which instructions are stored. When the instructions run on a computer, the computer is caused to execute some or all of the steps of the display method in the first aspect, the second aspect, all possible implementation manners of the first aspect, and all possible implementation manners of the second aspect.

[0019] Sixth aspect, embodiments of the present application provide a computer program product. When the computer program product runs on a computer, the computer is caused to execute some or all of the steps of the display method in the first aspect, the second aspect, all possible implementation manners of the first aspect, and all possible implementation manners of the second aspect. Description of the Drawings

[0020] To more clearly illustrate the technical solutions of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0021] Figure 1 It is a schematic diagram of a typical screen mirroring scenario provided by an embodiment of the present application;

[0022] Figure 2A It is a schematic diagram of an exemplary hardware structure of the first device 100 provided by an embodiment of the present application;

[0023] Figure 2B It is a schematic diagram of an exemplary software architecture of the first device 100 provided by an embodiment of the present application;

[0024] Figure 2C It is a schematic diagram of an exemplary architecture of the second device 200 provided by an embodiment of the present application;

[0025] Figure 3 It is a schematic diagram of an exemplary plane of the application display area (display) provided by an embodiment of the present application;

[0026] Figure 4A The first exemplary signaling interaction diagram of the display method provided by the embodiment of the present application;

[0027] Figure 4B The second exemplary signaling interaction diagram of the display method provided by the embodiment of the present application;

[0028] Figure 5A The first exemplary interface diagram of the screen mirroring scenario provided by the embodiment of the present application;

[0029] Figure 5B The second exemplary interface diagram of the screen mirroring scenario provided by the embodiment of the present application;

[0030] Figure 5C The third exemplary interface diagram of the screen mirroring scenario provided by the embodiment of the present application;

[0031] Figure 5C-1 Provided by the embodiment of the present application Figure 5C Exemplary diagram of the display corresponding to the scene shown;

[0032] Figure 5C-2 Provided by the embodiment of the present application Figure 5C An exemplary interface diagram of the mobile phone in the scene shown;

[0033] Figure 6 Exemplary interface diagram of the security interface provided by the embodiment of the present application;

[0034] Figure 7 Exemplary interface diagram of the first group of screen mirroring interfaces provided by the embodiment of the present application;

[0035] Figure 8 Provided by the embodiment of the present application Figure 7 Exemplary diagram of the display corresponding to the display interface;

[0036] Figure 9 Exemplary interface diagram of the second group of screen mirroring interfaces provided by the embodiment of the present application;

[0037] Figure 10 Provided by the embodiment of the present application Figure 9 Exemplary diagram of the display corresponding to the display interface;

[0038] Figure 11 Exemplary interface diagram of the third group of screen mirroring interfaces provided by the embodiment of the present application;

[0039] Figure 12 Exemplary interface diagram of the email login interface provided by the embodiment of the present application;

[0040] Figure 13 An exemplary schematic diagram of a display provided by an embodiment of the present application;

[0041] Figure 14A Provided by an embodiment of the present application Figure 13 The first exemplary screen mirroring interface schematic diagram of the display shown;

[0042] Figure 14B Provided by an embodiment of the present application Figure 13 The second exemplary screen mirroring interface schematic diagram of the display shown;

[0043] Figure 15 The third exemplary signaling interaction schematic diagram of the display method provided by an embodiment of the present application. Detailed implementation manners

[0044] Next, the technical solutions of the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application.

[0045] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular forms "a", "an", "the", "above-mentioned", "said", and "this" are also intended to include the plural forms, unless there is a clear indication to the contrary in the context. It should also be understood that although the terms first, second, etc. may be used in the following embodiments to describe a certain type of object, the object should not be limited to these terms. These terms are only used to distinguish specific objects of this type of object. For example, the terms first, second, etc. may be used in the following embodiments to describe devices, but the devices should not be limited to these terms. These terms are only used to distinguish different electronic devices. The same applies to other types of objects that may be described by the terms first, second, etc. in the following embodiments, and will not be repeated here.

[0046] An embodiment of the present application provides a display method, which is applied to a scenario where a first device screens at least two window interfaces to a second device. Among them, for the window interface without security requirements, the second device displays the content of the window interface, and for the window interface with security requirements, the second device displays a security interface. In this way, the second device can not only safely display the interface with security requirements, but also ensure that the user can normally operate the interface without security requirements.

[0047] The following introduces the first device, the second device, the user interface (UI) for such a first device and such a second device, and embodiments for using such a first device and such a second device.

[0048] To clearly describe the embodiments of the present application, in this specification, the interface displayed on the first device is referred to as the "window interface", and the "window interface" is a display interface with the size of the display screen of the first device as a window. In this specification, the interface displayed on the second device is referred to as the "screen mirroring interface", and each "screen mirroring interface" corresponds to a "window interface".

[0049] The first device involved in the embodiments of the present application may be an electronic device including a wireless communication module and a display function, such as a mobile phone, a tablet computer, a wearable device (e.g., a watch, a bracelet, a helmet, etc.), a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a smart home device (e.g., a smart TV, etc.) and other electronic devices. It can be understood that the specific type of the first device is not limited in the embodiments of the present application. The first device may be equipped with or a device with other operating systems.

[0050] Figure 2A FIG. shows a schematic hardware structure diagram of the first device 100. The first device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0051] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the first device 100. In some other embodiments of the present application, the first device 100 may include more or fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0052] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors. In some embodiments, the first device 100 may also include one or more processors 110.

[0053] Among them, the controller may be the nerve center and command center of the first device 100. The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of detection instructions, etc.

[0054] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the first device 100.

[0055] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0056] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be respectively coupled to the touch sensor 180K, the charger, the flashlight, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface to implement the touch function of the first device 100.

[0057] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple groups of I2S buses. The processor 110 may be coupled to the audio module 170 through the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 may transmit an audio signal to the wireless communication module 160 through the I2S interface to implement the function of listening to sound through a Bluetooth headset.

[0058] The PCM interface can also be used for audio communication to sample, quantize, and encode analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through the PCM bus interface. In some embodiments, the audio module 170 may also transmit an audio signal to the wireless communication module 160 through the PCM interface to implement the function of listening to sound through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0059] The UART interface is a general-purpose serial data bus for asynchronous communication. This bus can be a two-way communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface to implement the function of playing music through Bluetooth headphones.

[0060] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to implement the shooting function of the first device 100. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the first device 100.

[0061] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0062] The USB interface 130 is an interface that complies with the USB standard specification, and can specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the first device 100, and can also be used to transfer data between the first device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. This interface can also be used to connect other electronic devices, such as AR devices, etc.

[0063] It can be understood that the interface connection relationships between the modules illustrated in the embodiments of this application are only illustrative descriptions and do not constitute a structural limitation on the first device 100. In other embodiments, the first device 100 can also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0064] The charging management module 140 is configured to receive a charging input from a charger. The charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive the charging input from a wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 can receive the wireless charging input through the wireless charging coil of the first device 100. While charging the battery 142, the charging management module 140 can also supply power to the first device 100 through the power management module 141.

[0065] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives inputs from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as the battery capacity, the number of battery cycles, and the battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0066] The wireless communication function of the first device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0067] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the first device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as the diversity antenna of the wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0068] The mobile communication module 150 may provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc., applied to the first device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 may receive electromagnetic waves through the antenna 1, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 may also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least some functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be disposed in the same device.

[0069] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, receiver 170B, etc.) or displays the operation interface of the APP through the display screen 194. In some embodiments, the modulation and demodulation processor may be an independent device. In some other embodiments, the modulation and demodulation processor may be independent of the processor 110 and disposed in the same device as the mobile communication module 150 or other functional modules.

[0070] The wireless communication module 160 may provide solutions for wireless communications applied to the first device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSSs), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 for radiation.

[0071] In some embodiments, antenna 1 of the first device 100 is coupled to the mobile communication module 150, and antenna 2 is coupled to the wireless communication module 160, such that the first device 100 can communicate with the network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0072] In some embodiments, the wireless communication solution provided by the mobile communication module 150 enables the first device 100 to communicate with devices in the network (such as a cloud server). The WLAN wireless communication solution provided by the wireless communication module 160 also enables the first device 100 to communicate with devices in the network (such as the second device 200), so that the first device 100 can transmit a video stream to the second device 200, realizing the function of casting the screen of the first device 100 to the second device 200.

[0073] The first device 100 implements the display function through a GPU, a display screen 194, an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. For example, the GPU can render an image as a security interface. The processor 110 may include one or more GPUs, which execute program instructions to generate or change the interface display effect. In the embodiments of the present application, the display screen 194 may include a display and a touch control device. The display is used to output display content to the user, such as the desktop page of the first device 100 and various operation interfaces of the APP. The touch control device is used to receive operation instructions input by the user on the display screen 194.

[0074] The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc.

[0075] In some embodiments of the present application, when the display panel adopts materials such as OLED, AMOLED, FLED, etc., the display screen 194 can be bent. Here, the display screen 194 can be bent means that the display screen can be bent to any angle at any part and can be held at that angle. For example, the display screen 194 can be folded in half left and right from the middle. It can also be folded in half up and down from the middle.

[0076] The first device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, etc.

[0077] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light passes through the lens and is transmitted to the camera sensor. The optical signal is converted into an electrical signal, and the camera sensor transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise, brightness, and skin color of the image through algorithms. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0078] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard formats such as RGB and YUV.

[0079] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the first device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0080] The video codec is used to compress or decompress digital videos. In some embodiments, after the first device 100 is connected to the second device 200, the video codec can compress the video stream to be projected onto the second device 200, and then send the compressed video file to the second device 200. The first device 100 can support one or more video codecs.

[0081] The NPU is a neural-network (NN) computing processor. By learning from the structure of biological neural networks, such as the transmission mode between human brain neurons, it can quickly process the input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the first device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0082] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the first device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to achieve the data storage function. For example, data such as music, photos, and videos are saved in the external memory card.

[0083] The internal memory 121 can be used to store one or more computer programs, and the one or more computer programs include instructions. The processor 110 can execute the above instructions stored in the internal memory 121, so that the first device 100 executes the display method provided in some embodiments of the present application, as well as various functional applications and data processing, etc. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store the operating system; the program storage area can also store one or more application programs (such as email, gallery, contacts, etc.). The data storage area can store the data created during the use of the first device 100 (such as window interface identifiers, etc.). In addition, the internal memory 121 can include high-speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (UFS), etc.

[0084] The first device 100 can implement audio functions through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor, etc. For example, music playback, recording, etc.

[0085] The audio module 170 is used to convert digital audio information into analog audio signals for output, and is also used to convert analog audio inputs into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0086] The speaker 170A, also known as the "loudspeaker", is used to convert audio electrical signals into sound signals. The first device 100 can listen to music or hands-free calls through the speaker 170A.

[0087] The receiver 170B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. When the first device 100 answers a call or voice message, the voice can be listened to by bringing the receiver 170B close to the ear.

[0088] The microphone 170C, also known as the "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak close to the microphone 170C with their mouth to input the sound signal into the microphone 170C. The first device 100 can be provided with at least one microphone 170C. In some other embodiments, the first device 100 can be provided with two microphones 170C, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the first device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, and implement functions such as directional recording.

[0089] The headphone jack 170D is used to connect a wired headphone. The headphone jack 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0090] The pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can include at least two parallel plates with conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The first device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the first device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The first device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations with the same touch position but different touch operation intensities can correspond to different operation instructions.

[0091] The gyroscope sensor 180B can be used to determine the motion posture of the first device 100. In some embodiments, the angular velocity of the first device 100 around three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. Exemplarily, when the shutter is pressed, the gyroscope sensor 180B detects the shaking angle of the first device 100, calculates the distance that the lens module needs to compensate according to the angle, and makes the lens offset the shaking of the first device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used in navigation and somatosensory game scenarios.

[0092] The barometric pressure sensor 180C is used to measure barometric pressure. In some embodiments, the first device 100 calculates the altitude based on the barometric pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0093] The magnetic sensor 180D includes a Hall sensor. The first device 100 can use the magnetic sensor 180D to detect the orientation of the display screen 194 of the first device 100. In some embodiments, when the first device 100 is a flexible screen device, the first device 100 can detect the folding and unfolding states of the screen according to the magnetic sensor 180D.

[0094] The acceleration sensor 180E can detect the magnitude of the acceleration of the first device 100 in various directions (generally three axes). When the first device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the first device 100 and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0095] The distance sensor 180F is used to measure distance. The first device 100 can measure distance by infrared or laser. In some embodiments, when shooting a scene, the first device 100 can use the distance sensor 180F to measure distance to achieve rapid focusing.

[0096] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The light-emitting diode can be an infrared light-emitting diode. The first device 100 emits infrared light outward through the light-emitting diode. The first device 100 uses the photodiode to detect the infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the first device 100. When insufficient reflected light is detected, the first device 100 can determine that there is no object near the first device 100. The first device 100 can use the proximity light sensor 180G to detect when the user holds the first device 100 close to the ear for a call, so as to automatically turn off the screen to achieve the purpose of power saving. The proximity light sensor 180G can also be used for automatic unlocking and locking in the leather case mode and pocket mode.

[0097] The ambient light sensor 180L is used to sense the ambient light brightness. The first device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the first device 100 is in the pocket to prevent accidental touch.

[0098] The fingerprint sensor 180H is used to collect fingerprints. The first device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering of incoming calls, etc.

[0099] The temperature sensor 180J is used to detect temperature. In some embodiments, the first device 100 executes a temperature processing strategy by using the temperature detected by the temperature sensor 180J. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the first device 100 reduces the performance of the processor near the temperature sensor 180J to reduce power consumption and implement thermal protection. In some other embodiments, when the temperature is lower than another threshold, the first device 100 heats the battery 142 to prevent abnormal shutdown of the first device 100 caused by low temperature. In some other embodiments, when the temperature is lower than yet another threshold, the first device 100 boosts the output voltage of the battery 142 to prevent abnormal shutdown caused by low temperature.

[0100] The touch sensor 180K, also known as a touch panel or a touch-sensitive surface. The touch sensor 180K can be disposed on the display screen 194, and together with the display screen 194, it forms a touch screen, also known as a "touch display screen". The touch sensor 180K is used to detect touch operations acting thereon or nearby. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K can also be disposed on the surface of the first device 100, at a different position from the display screen 194.

[0101] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire the vibration signals of the vibrating bone mass of the human vocal part. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulsation signals. In some embodiments, the bone conduction sensor 180M can also be disposed in the earphone to form a bone conduction earphone. The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bone mass of the human vocal part acquired by the bone conduction sensor 180M to implement the voice function. The application processor can parse out heart rate information based on the blood pressure pulsation signals acquired by the bone conduction sensor 180M to implement the heart rate detection function.

[0102] The keys 190 include a power-on key, volume keys, etc. The keys 190 can be mechanical keys or touch keys. The first device 100 can receive key inputs to generate key signal inputs related to the user settings and function control of the first device 100.

[0103] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and also for touch vibration feedback. For example, touch operations for different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. For touch operations on different areas of the display screen 194, the motor 191 can also correspond to different vibration feedback effects. Different application scenarios (such as time reminder, receiving messages, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0104] The indicator 192 can be an indicator light and can be used to indicate the charging status, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0105] The SIM card interface 195 is used to connect the SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation from the first device 100. The first device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The first device 100 interacts with the network through the SIM card to implement functions such as calls and data communication. In some embodiments, the first device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the first device 100 and cannot be separated from the first device 100.

[0106] The software system of the first device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In this embodiment of the application, the Android system with a layered architecture is taken as an example to exemplarily illustrate the software structure of the first device 100.

[0107] Figure 2B is a schematic diagram of the software structure of the first device 100 in this embodiment of the application.

[0108] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system from top to bottom is respectively the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0109] The application layer can include a series of application packages.

[0110] Such as Figure 2BAs shown, the application package may include applications such as a gallery, a calendar, a map, WLAN, music, text messages, calls, navigation, Bluetooth, video, etc.

[0111] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0112] Such as Figure 2B As shown, the application framework layer may include a display manager service (DMS), an activity manager service (AMS), and a window manager service (WMS). The window manager service can also be called a window manager. Of course, the application framework layer may also include a content provider, a view system, a phone manager, a resource manager, a notification manager, etc., and the embodiments of the present application do not make any restrictions on this.

[0113] Among them, the WMS can be used to determine the display policy of the APP in the first device 100. After that, the WMS can control the APP to be displayed on the display screen 194 of the first device 100 according to the display policy through the display driver and the surface manager of the underlying display system.

[0114] Among them, the DMS can be used to configure the display policy of the window interface to be cast to the second device 200 within the application display area (display) of the first device 100, so that the second device 200 can display the corresponding cast screen interface based on the display policy of the window interface within the display.

[0115] Among them, the display is the display area of the application running on the first device. From the perspective of screen mirroring, it can also be understood as the presentation area configured by the WMS for the window interface to be cast. The WMS can configure the display according to the area size occupied by the window interface to be cast when tiled. Such as Figure 3 As shown, the WMS can use the upper left corner of the first device 100 as the origin of the coordinate system and the width of the display screen 194 of the first device 100 as the unit to configure the aforementioned display. Exemplarily, the WMS can be extended in the x-axis direction to obtain the aforementioned display. Among them, the size of the display is associated with the number of window interfaces to be cast. For example, Figure 3In the embodiment illustrated in (a), when the window interface to be cast is a certain scenario, the display area of the display screen 194 can accommodate the window interface to be cast, and the WMS configures the display area of the display screen 194 as display30. For another example, Figure 3 In the embodiment illustrated in (b), when the window interfaces to be cast are two, the display area of the display screen 194 cannot accommodate the two window interfaces to be tiled and displayed. Based on this, the WMS expands a display area with the width of the display screen 194 in the x-axis direction. Furthermore, the WMS configures one window interface to be cast to be displayed in the display area of the display screen 194, and the other window interface to be cast to be displayed in the expanded area. That is, Figure 3 In the embodiment illustrated in (b), display31 includes a first display311 and a second display312. The first display311 is configured on the display screen 194 of the first device 100 and is a display visible to the user. The second display312 is an extended display of the first display311 and is a display invisible to the user. In the embodiment of the present application, this extended display is also referred to as a "virtual display". It should be understood that the size of the virtual display is associated with the number of window interfaces to be cast. For example, when the window interfaces to be cast are three, the WMS expands two display areas with the width of the display screen 194 in the x-axis direction. Correspondingly, the width of the virtual display is two times the width of the display screen 194.

[0116] The AMS can be used to manage the running processes of the APP. The running processes of the APP include the start, pause, and termination of the APP process, etc. Among them, an APP usually consists of multiple Activities. Specifically, the AMS is used to manage different Activities of the APP in different processes of the APP. For example, when the login interface of the email APP is running, the AMS calls a corresponding Activity of the email APP. In the process of displaying the main interface of the email, the AMS calls another corresponding Activity of the email APP.

[0117] The AMS can also be used to set a task identity (task identity, task ID) for the launched APP after an APP process is started, so that all Activities of the APP correspond to the set task ID, facilitating the first device 100 to distinguish each window interface and the second device 200 to distinguish each screen mirroring interface. Exemplarily, the AMS can set task IDs for the APPs according to the number of APPs running on the first device 100. For example, the AMS can set the task identity of the main interface of the first device 100 to task0. After the process of the mailbox APP on the first device 100 is started, the AMS sets the task identity "task1" for the mailbox APP. In some other embodiments, the AMS can also be used to delete the task identity corresponding to the corresponding APP after an APP process is terminated.

[0118] The WMS can be used to manage window programs. For example, the WMS can obtain the size of the display screen, the size of the display, and the positions and coordinates of each window interface, etc. Additionally, the WMS can also traverse whether the current interface of the running APP contains a security flag bit (implemented by viewing the parameters of windowstate). After traversing that the current interface (Activity) of the APP contains the security flag bit, it triggers the first device 100 to send a security protection instruction to the second device 200, so that the second device 200 displays the security interface at the forefront of the screen mirroring interface of the APP window interface, and at the same time terminates the screen mirroring of the display where the application is located to the second device 200. The security protection instruction may include, for example, the task identity of the corresponding window interface.

[0119] In some embodiments, the security flag bit can be implemented as a security flag (flag), for example. In actual implementation, the security flag can include multiple types of flags, and different types of flags can indicate different types of security requirement scenarios. For example, the first flag can indicate an application security lock. In actual operation, when an application with a security lock is opened upon receiving a user operation, it triggers the security mode of the application security lock. The second flag can be actively set by the application. In some embodiments, biometric identification (such as face unlocking), or implementation scenarios such as a display interface containing privacy information (such as an ID card photo), trigger the security mode actively set by the application. In some other embodiments, some applications set that the current Activity cannot be screen captured or recorded, triggering the security mode actively set by the application. The third flag can indicate the invocation of a secure keyboard. In actual operation, implementation scenarios such as the input of bank card account numbers and / or passwords trigger the security mode of invoking the secure keyboard. It can be understood that in some embodiments, the flag is associated with the Activity, that is, one Activity corresponds to one flag.

[0120] In some embodiments, determining whether a flag exists as referred to in the present application can be understood as judging the value of the flag. For example, if the flag is 1, it is judged to exist; if the flag is 0, it is judged not to exist. Another example is that if the flag is 0, it is judged to exist; if the flag is 1, it is judged not to exist. The present application does not limit this.

[0121] Based on the description of the foregoing embodiments, a window interface can correspond to multiple Activities. During the process of calling an Activity of the window interface, the Window state of the corresponding APP is updated. In some embodiments, during the implementation of updating the Window state, at least one of the foregoing flags can be updated. Based on this, the WMS can determine whether the window interface contains a security flag bit by traversing whether the Windowstate of the window interface contains the flag. If the Windowstate corresponding to the corresponding Activity contains any of the foregoing flags, it indicates that the current window interface of the APP contains a security flag bit, and the first device 100 then sends a security protection instruction to the second device 200.

[0122] The content provider is used to store and obtain data, and make this data accessible to application programs. The data can include videos, images, audio, dialed and answered calls, browsing history and bookmarks, phone books, etc.

[0123] The view system includes visible controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build application programs. The display interface can be composed of one or more views. For example, a display interface including a camera icon.

[0124] The phone manager is used to provide the communication function of the first device 100. For example, the management of call states (including connection, hanging up, etc.).

[0125] The resource manager provides various resources for application programs, such as localized strings, icons, pictures, layout files, video files, etc.

[0126] The notification manager enables application programs to display notification information in the status bar. It can be used to convey message types that can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform that the download is completed, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as a notification of a background running application program, or a notification that appears on the screen in the form of a dialogue window. For example, prompting text information in the status bar, emitting a prompt sound, the first device 100 vibrating, the indicator light flashing, etc.

[0127] The Android Runtime includes the core libraries and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0128] The core libraries consist of two parts: one is the functional functions that the Java language needs to call, and the other is the core libraries of Android.

[0129] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as the management of object life cycles, stack management, thread management, security and exception management, and garbage collection.

[0130] Continue to refer to Figure 2B , the system libraries and the kernel layer below the application framework layer can be called the underlying system. The underlying system includes the underlying display system for providing display services. For example, the underlying display system includes the display driver in the kernel layer and the surface manager in the system libraries, etc.

[0131] The system libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (such as: OpenGL ES), 2D graphics engines (such as: SGL), etc.

[0132] The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0133] The media libraries support the playback and recording of various common audio and video formats, as well as static image files, etc. The media libraries can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0134] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0135] The 2D graphics engine is the graphics engine for 2D drawing.

[0136] The kernel layer is the layer between the hardware and the software. The kernel layer at least includes the display driver, the camera driver, the audio driver, and the sensor driver.

[0137] The second device involved in the embodiments of the present application can be an electronic device including a wireless communication module and a display function, such as electronic devices like monitors, smart TVs, tablets, laptops, UMPCs, netbooks, etc. It can be understood that the embodiments of the present application do not impose any restrictions on the specific type of the second device. The second device can be equipped with or a device with other operating systems.

[0138] Figure 2C Fig. shows an exemplary schematic architecture of the second device 200. The second device 200 may include a processor 210, a memory 220, a wireless communication module 230, a display screen 240, a power supply 250, etc.

[0139] It can be understood that the structure illustrated in this application does not constitute a specific limitation on the second device 200. In other embodiments of this application, the second device 200 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0140] In this embodiment, the hardware and software included in the processor 210 and the functions of each hardware and software are similar to those of the processor 110. Details are not described here again in this application. The processor 210 may be used to call the display screen 240 to display a screen mirroring interface and a security interface, etc. from the first device 100. In some embodiments, the processor 210 may also be used to generate operation instructions for instructing the user to operate the APP on the first device 100 side.

[0141] The communication solution provided by the wireless communication module 230 is similar to that of the wireless communication module 160, and details are not described here again. The communication solution provided by the wireless communication module 230 enables the second device 200 to communicate with the first device 100, so as to realize the interaction functions of video streams and instructions between the second device 200 and the first device 100. For example, the wireless communication module 230 realizes the function of the second device 200 receiving the video stream to be screen mirrored from the first device 100.

[0142] The memory 220 may be used to store one or more computer programs, and the one or more computer programs include instructions. The processor 210 may execute the above instructions stored in the memory 220, so that the second device 200 executes the display method and the like provided in some embodiments of this application. The memory 220 may also be used to cache data involved in the process of the second device 200 executing the display method, such as data of the security interface, etc.

[0143] The display screen 240 is used to display the controls, information, images, videos, etc. of the second device 200, as well as the screen mirroring interface information from the first device 100. In some other embodiments, when the second device 200 receives a security protection instruction from the first device 100, it can trigger the display screen 240 to display a security interface at the forefront of the screen mirroring interface corresponding to the security protection instruction. In this embodiment, the hardware composition of the display screen 240 is similar to that of the aforementioned display screen 194, which will not be elaborated here.

[0144] The power supply 240 can be used to supply power to the processor 210, the memory 220, the display screen 240, etc.

[0145] The UI for implementing the display function involved in the embodiments of this application is a media interface for interaction and information exchange between an application or an operating system and a user, and it realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on the terminal device and finally presented as content recognizable by the user, such as controls like videos, pictures, texts, buttons, etc. A control (also known as a widget) is a basic element of the user interface. Typical controls include a toolbar, a menu bar, a text box, a button, a scrollbar, pictures, and texts. The attributes and content of the controls in the interface are defined through tags or nodes. For example, XML passes through <textview> 、 <imgview> 、 <videoview>Such nodes are used to define the controls included in the interface. One node corresponds to one control or property in the interface, and after being parsed and rendered, the node presents visible content to the user. In addition, in the interfaces of many applications, such as hybrid applications, there are usually web pages included. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hyper text markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as recognizable content to the user by a browser or a web page display component similar to the browser function. The specific content included in the web page is also defined by tags or nodes in the web page source code. For example, HTML uses 、 、 <video> 、 <canvas>Define the elements and attributes of a web page.

[0146] A commonly used form of the user interface is the graphical user interface (GUI), which refers to the user interface related to computer operations displayed in a graphical manner. It can be an interface element such as an icon, window, control, etc. displayed on the display screen of an electronic device, where the control can include visible interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, Widgets, etc.

[0147] It should be noted that in a typical security protection mechanism based on the screen mirroring scenario, the first device 100 mirrors the screen to the second device 200, usually in units of display for screen mirroring, and the second device 200 displays the security interface in units of window interfaces. Based on this, if the display mirrored by the first device 100 contains at least two window interfaces, the second device 200 can correspondingly display at least two screen mirroring interfaces. When any one of the at least two window interfaces requires security protection, the second device 200 displays the security interface in units of window interfaces, so that all of the at least two screen mirroring interfaces display the security interface or a black screen (at this time, the display data of the first device will not be transmitted to the second device). In this way, the user cannot operate the screen mirroring interface without security requirements normally, and the user experience is poor.

[0148] To solve the above problems, the embodiments of the present application provide a display method and an electronic device. As Figure 4A shown, the display method involved in the embodiments of the present application includes: The first device receives a screen mirroring instruction input by the user. In response to the screen mirroring instruction, the first device mirrors the first window interface to the second device. Correspondingly, the second device displays the screen mirroring interface of the first window interface. Furthermore, after the first device runs the second window interface in response to the instruction, it mirrors the second window interface to the second device (this instruction comes from the second device. For example, when the first window interface is the main interface, this instruction is an operation of double-clicking to run an application). The first device traverses the Window state and then confirms that the second window interface contains a security flag bit, and sends the task number and security protection instruction of the second window. The second device displays the screen mirroring interface and the security interface of the first window interface (the security interface is used to replace the original second window interface for display).

[0149] Among them, the trigger for the first device to traverse the Window state can be a periodic trigger, or it may be triggered by a change in the position of the Activity in the stack, or it may be triggered by a change in the Activity at the top of the stack.

[0150] Based on this, Figure 4A In the illustrated implementation scenario, after the first device starts the second window interface, the flag corresponding to the Activity of the second window interface presented in response to the start instruction is 1. For example, if the second window interface is the login interface of a banking APP, a secure keyboard is triggered after the second window is started. In this way, before mirroring the screen, the first device can immediately determine that the second window contains a security flag. Furthermore, the first device sends the task number and security protection instruction of the second window to the second device, so that the second device can display the mirrored screen interface and the security interface of the first window interface.

[0151] In some other embodiments, the embodiments of the present application also provide another exemplary display method. As Figure 4A shown, in this embodiment, the first device receives a screen mirroring instruction input by the user. In response to this screen mirroring instruction, the first device mirrors the first window interface to the second device. After the first device runs the second window interface in response to the instruction, it determines that the second window interface does not contain a security flag bit during the first traversal cycle. Furthermore, it mirrors the second window interface to the second device. Subsequently, in subsequent traversal cycles, the first device detects that the second window interface contains a security flag bit. Then, the first device sends the task number and security protection instruction of the second window interface to the second device. Correspondingly, the second device displays the mirrored screen interface and the security interface of the first window interface.

[0152] Figure 4B In the illustrated implementation scenario, after the first device starts the second window interface, the content of the currently started second window interface has no security requirements. Correspondingly, the flag of the current Window state of the second window interface is set to 0. Then, the first device can mirror the content of this second window interface to the second device, so that the second device can display the mirrored screen interface of the first window interface and the mirrored screen interface of the second window interface. In response to the user's trigger, the second window interface changes, which triggers a change in the Activity of the changed interface, causing the changed second window interface to have security requirements. Correspondingly, the flag of the Window state is set to 1. Furthermore, the first device can determine that the second window interface at this time contains a security flag bit, and the first device triggers the second device to display the second window interface as a security interface, so that the second device can display the mirrored screen interface of the first window interface and the security interface.

[0153] For example, after the second window interface is launched, the main interface of the wallet APP is displayed. If the main interface of the wallet APP, for example, contains photos without security requirements, then the first device projects the wallet APP onto the second device. The second device can display the projected interface of the main interface of the wallet APP. Furthermore, in response to the user's trigger, the second window interface enters the payment interface of the wallet APP. At this time, the second window interface includes a payment password input control and is an interface with security requirements. After detecting the security flag bit, the first device sends the task number of the second window interface (the interface of the wallet APP) and a security protection instruction to the second device, so that the second device displays the projected interface of the second window interface as a secure interface.

[0154] Furthermore, in the above Figure 4A or Figure 4B In the illustrated embodiment, if the second window interface further updates the displayed content in response to the user's operation instruction and the corresponding content has no security requirements, the flag of the corresponding Window state is set to 0. Then, during the periodic traversal of the Window state by the first device, the first device can determine that the second window interface has no security flag bit based on the Window state. Furthermore, the first device can project the updated second window interface onto the second device. Correspondingly, the second device hides the secure interface and displays the projected interface of the updated second window interface.

[0155] According to the foregoing description of projection, the content projected by the first device to the second device is the window interface included in the display of the first device, that is, the first window interface and the second window interface are the window desktops included in the display of the first device. It should be understood that if the display of the first device also includes a third window interface, a fourth window interface, etc., the first device should also detect whether the third window interface and the fourth window interface include a security flag bit. After that, the first device sends the projection information of the third window interface and the fourth window interface to the second device according to the detection result. This is not elaborated here. In addition, in the embodiments of the present application, every time the display of the first device is updated, the first device performs a projection. Among them, the addition or reduction of the window interfaces in the display, as well as any change to any window interface, are all considered as an update of the display.

[0156] It should be noted that screen mirroring from the first device to the second device can be implemented as follows: after establishing a network connection between the first device and the second device, the first device sends the interface to be displayed to the second device in the form of a video stream. Subsequently, the second device can cut the video stream based on the parameter information sent by the first device (such as the size of the window interface, the position of the window interface, and the relative position between window interfaces), and then display each interface mirrored from the first device in the form of a window. Based on this, in the implementation scenario of screen mirroring, the WMS is configured to tile and display the displays of the windows to be mirrored, and the AMS adds a task ID to each window interface to be mirrored, so that the first device can generate a video stream corresponding to each window interface to be mirrored and identify each window interface to be mirrored.

[0157] It should be noted that the display contains the window interfaces of all the APPs running on the first device, and the windows to be mirrored can be all the window interfaces in the display or some of the window interfaces in the display.

[0158] In some embodiments, the first device determines whether each window interface has security requirements before screen mirroring. Then, based on the detection results, the first device sends the task ID of each window interface and the screen mirroring information of this window interface to the second device. The second device displays the security interface and the screen mirroring interface containing the content of the window interface based on the task ID, where the security interface is the screen mirroring interface of the window interface with security requirements. In this way, the second device can not only safely display the interfaces with security requirements but also ensure that the user can operate the interfaces without security requirements normally.

[0159] For ease of understanding, in the following embodiments of the present application, a mobile phone with the Figure 2A and Figure 2B shown structure will be taken as an example, and a laptop computer with the Figure 2C shown structure will be taken as an example to specifically elaborate on the display method in the screen mirroring scenario provided by the embodiments of the present application in combination with the accompanying drawings.

[0160] The embodiments of the present application can divide the actual implementation process into the following three stages: the stage of starting the screen mirroring function, the stage of display in the security requirement scenario, and the stage of display after operating on the security interface. The embodiments of the display method will be introduced below in combination with these three stages.

[0161] The stage of starting the screen mirroring function:

[0162] In the embodiments of the present application, the mobile phone and the laptop can establish a communication connection through their respective wireless communication modules. After receiving the instruction to turn on the screen mirroring function input by the user, the mobile phone can, for example, send the video stream to be screen mirrored to the laptop based on the Wi-Fi Direct technology to implement the screen mirroring function. In some embodiments, the mobile phone and the laptop can establish a communication connection through their respective NFC modules or Bluetooth modules, for example. After receiving the instruction for the user to click to turn on "multi-device collaboration", the screen mirroring function of the mobile phone is turned on. In a scenario where the distance between the mobile phone and the laptop is less than or equal to the communication transmission distance, the mobile phone screens the first set of window interfaces to the laptop. If the above-mentioned screen mirroring scenario requirements are met, it is considered that the mobile phone has received the screen mirroring instruction. Details are not described here. The display interface related to the mobile phone screening the first set of window interfaces to the laptop is as Figures 5A to 5C-2 shown.

[0163] It can be understood that the modules for the mobile phone and the laptop to establish a communication connection and the interfaces for turning on the screen mirroring function of the mobile phone are all schematic descriptions and do not constitute a limitation to the embodiments of the present application. In other embodiments, the mobile phone and the laptop can also establish a connection based on other communication modules. In addition, the interfaces for turning on the screen mirroring function of the mobile phone can vary according to the model of the mobile phone and the version of the mobile phone operating system, which is not limited here.

[0164] As Figure 5A shown, Figure 5A the schematic GUI51 is the first exemplary display interface of the laptop after the mobile phone is screen mirrored to the laptop. Among them, the GUI51 includes a screen mirroring interface 510, which is the screen mirroring interface of the mobile phone desktop and includes images of various interface elements in the mobile phone desktop, such as the gallery, email, wallet, etc.

[0165] Figure 5A In the corresponding embodiment, during the reception of the screen mirroring instruction, regardless of what interface the display screen of the mobile phone is showing and regardless of what APP is running in the foreground of the mobile phone, the mobile phone calls the wireless communication module to send the video stream of the mobile phone desktop to the laptop. Furthermore, the laptop decodes the corresponding video stream and displays the screen mirroring interface 510 corresponding to the video stream. At this time, the APPs running on the mobile phone optionally all run in the form of free floating windows.

[0166] In some embodiments, the aspect ratio of the screen mirroring interface may be the same as that of the window interface. The size of the screen mirroring interface depends on the size of the screen of the second device. For example, if the size of the screen of the second device is 285.7 millimeters (mm) wide * 214.3 mm high, the size of the screen mirroring interface may be 110 mm wide * 210 mm high. If the size of the screen of the second device is 1456.4 mm wide * 850.9 mm high, the size of the screen mirroring interface may be 450 mm wide * 800 mm high.

[0167] As Figure 5B shown, Figure 5B the schematic GUI52 is an exemplary display interface of the mobile phone during the screen mirroring process. Figure 5B The schematic GUI53 is the second exemplary display interface of the laptop after the mobile phone is screen-mirrored to the laptop. Among them, GUI52 is, for example, the display interface of the Gallery APP running on the mobile phone. GUI52 includes image 521 and image 522. GUI53 includes the screen mirroring interface 530. The screen mirroring interface 530 is the screen mirroring interface of GUI52. The screen mirroring interface 530 includes the screen mirroring image 531 of image 521 and the screen mirroring image 532 of image 532.

[0168] Figure 5B In the corresponding embodiment, during the reception of the screen mirroring instruction, the mobile phone is displaying the window interface. Furthermore, the WMS traverses whether the Activity corresponding to the window interface contains a flag. If the Activity corresponding to the window interface does not contain a flag, the GPU of the mobile phone can render the interface video stream that the mobile phone is currently displaying. After that, the mobile phone calls the wireless communication module to send the video stream of the interface that the mobile phone is currently displaying to the laptop. Furthermore, the laptop decodes the corresponding video stream and displays the screen mirroring interface 530 corresponding to the video stream.

[0169] In some other embodiments, if the Activity corresponding to the window interface contains a flag, the mobile phone can screen-mirror the desktop of the mobile phone to the laptop. Correspondingly, the GUI of the laptop can be as shown in GUI51. After the mobile phone receives the user's operation and updates the window interface, the mobile phone can screen-mirror the updated window interface to the laptop. For details, see the description of the relevant embodiments below, and details are not described here.

[0170] In some other embodiments, if the mobile phone does not display the interface of any APP, the mobile phone screen-mirrors the desktop of the mobile phone to the laptop. Correspondingly, the GUI of the laptop can be as shown in GUI51.

[0171] As Figure 5C shown, Figure 5C the schematic GUI54 is another exemplary display interface of the mobile phone during the screen mirroring process. Figure 5C The schematic GUI 55 is the third exemplary display interface of the laptop computer after the mobile phone is screen-cast to the laptop computer. Among them, GUI 54 is, for example, the display interface of the window interface where the mobile phone is running and has security requirements. For an exemplary display interface of GUI 54, see Figure 5C-2 as shown, which will not be elaborated here. GUI 55 includes a screen-cast interface 551 and a screen-cast interface 552. Among them, the screen-cast interface 551 is the screen-cast interface of the window interface where the mobile phone is running and has no security requirements, and the screen-cast interface 552 is the screen-cast interface of the mobile phone desktop.

[0172] Figure 5C In the corresponding embodiment, during the reception of the screen-cast instruction, the mobile phone is, for example, running several APPs. In this embodiment, the mobile phone can respectively detect whether the window interfaces where the several APPs are running contain a security flag bit. After that, the mobile phone can screen-cast the window interfaces that do not contain the security flag bit and the desktop of the mobile phone to the laptop computer, and stack and display the window interfaces that contain the security flag bit (such as Figure 5C GUI 54 shown) on the display screen of the mobile phone, so as to facilitate the user to perform the next operation. And the laptop computer displays the screen-cast interface of the window interface without security requirements (such as Figure 5C GUI 55 shown).

[0173] Exemplarily, during the reception of the screen-cast instruction, the mobile phone is, for example, running three APPs: email, wallet, and gallery. Among them, the current window interface of the email is, for example, the login interface, and the current window interface of the wallet is, for example, the interface for managing bank cards. Furthermore, the WMS traverses the Activities corresponding to the three window interfaces to determine whether each Activity contains a flag. For example, the WMS determines that the Activity corresponding to the current window interface of the email contains a first flag, the Activity corresponding to the current window interface of the wallet contains a second flag, and the Activity corresponding to the current window interface of the gallery does not contain a flag. After that, the WMS can report the traversal result to the DMS. The DMS configures the display of the mobile phone and the display strategy of each window interface based on the traversal result. The display interface of the said display is as Figure 5C-1 shown.

[0174] As Figure 5C-1 shown, Figure 5C-1 The schematic display 50 includes display 501 and display 502. Among them, display 501 is deployed on the display screen of the mobile phone and is the display visible to the user. Display 502 is an extended display of display 501 and is a virtual display invisible to the user. In this embodiment, there is no security requirement for the desktop and gallery window interfaces of the mobile phone. Therefore, the mobile phone can project the desktop and gallery window interfaces to the laptop. Based on this, AMS can configure task IDs for the desktop and gallery window interfaces of the mobile phone respectively. For example, AMS configures task 0 for the desktop of the mobile phone and task 1 for the gallery window interface. Moreover, DMS can configure the desktop of the mobile phone to be presented on display 501 and configure the gallery window interface to be presented on display 502. Furthermore, the GPU of the mobile phone can render the video streams of the desktop and gallery window interfaces of the mobile phone. After that, the mobile phone calls the wireless communication module to send task 0 and the video stream of the mobile phone desktop, as well as task 1 and the video stream of the gallery window interface to the laptop. Furthermore, the laptop decodes the corresponding video streams, splits the decoded images, and then displays Figure 5C the GUI 55 in it. The screen mirroring interface 551 in the GUI 55 is, for example, the screen mirroring interface of the mobile phone desktop, and the screen mirroring interface 551 corresponds to task 0. The screen mirroring interface 552 is, for example, the screen mirroring interface of the gallery window interface, and the screen mirroring interface 552 corresponds to task 1.

[0175] Furthermore, the window interfaces of the mailbox and the wallet are interfaces with security requirements. Even if they are projected to the laptop, the laptop is not allowed to display the contents of these two window interfaces. Based on this, the mobile phone can refrain from projecting these two window interfaces to the laptop and instead display these two window interfaces on the display screen of the mobile phone. Among them, DMS can configure the display strategies of these two window interfaces on the display screen of the mobile phone based on the traversal results of WMS. After that, DMS can display these two window interfaces on the display screen of the mobile phone through the display driver and the surface manager. The display interfaces of these two window interfaces are as Figure 5C-2 shown.

[0176] Figure 5C-2 The schematic GUI56 is an exemplary display interface of GUI54. GUI56 is a display interface for the image of window interface 561 and the image of window interface 562. The images of window interface 561 and window interface 562 are stacked and displayed, and the bottom end of GUI56 further includes a delete icon 563, etc. Window interface 561 is, for example, the login interface of an email box, and window interface 561 is, for example, the interface for managing bank cards in a wallet. After receiving the instruction that the user clicks the delete icon 563, AMS ends the processes of the email box and the wallet. After receiving the instruction that the user clicks the image of window interface 561, DMS controls the login interface of the email box to be displayed at the forefront of the mobile phone display screen, so that the user can input the account number and password of the email box. Similarly, after receiving the instruction that the user clicks the image of window interface 562, DMS controls the interface for managing bank cards in the wallet to be displayed at the forefront of the mobile phone display screen, so that the user can perform the next operation, such as a payment operation, etc. Details are not described herein again.

[0177] In some embodiments, the mobile phone can display GUI56 after Figure 5C-1 casting the schematic display50 to a laptop computer.

[0178] Furthermore, after the mobile phone casts display50 to the laptop computer, the mobile phone displays GUI56 and other subsequent displays, which have triggered the update of the window interface. Correspondingly, display50 is triggered to be updated to another display. Then, WMS traverses whether the window interfaces in the updated display contain security flag bits. Furthermore, the mobile phone can perform screen mirroring on the updated display according to the detection result.

[0179] Adopting this implementation method, after the mobile phone receives the screen mirroring instruction, regardless of whether the window interface of the currently running APP has security requirements, the mobile phone selects a group of interfaces without security requirements as the first group of window interfaces to be cast to the laptop computer. In this way, it can not only cast at least one operable interface to the laptop computer so that the user can continue to operate, but also ensure the security of the window interfaces with security requirements, and can save transmission resources.

[0180] It should be noted that in some possible implementation schemes, the mobile phone can send the security interface to the laptop computer at this stage. Correspondingly, the laptop computer can store the security interface locally, and after receiving the security protection instruction from the mobile phone, display the screen mirroring interface indicated by the security protection instruction as the security interface.

[0181] Figure 6 Schematically shows a security interface 60. The security interface 60 includes an icon 61 and an icon 62. The icon 61, for example, displays the following information: "Operation is required on the mobile phone side to protect your privacy". The icon 62, for example, displays the following information: "Display this application on the mobile phone". The icon 62 optionally receives a click from the user on the computer and is sent to the mobile phone side through, for example, the Miracst UIBC (user input back control) protocol. The background color of the security interface 60 can be a solid color, such as black. In some other embodiments, in order to optimize the user's visual experience, the background of the security interface 60 can be a Gaussian-blurred mobile phone wallpaper.

[0182] In some embodiments, the background of the security interface is optionally generated by the computer side. In some embodiments, in order to better provide a consistent experience for the user on the mobile phone and the computer, the security interface is optionally generated by the mobile phone side. For example, the mobile phone performs Gaussian blur processing on its wallpaper and then sends it to the computer, and the computer stores the processed wallpaper. When the computer receives the display of the security interface sent by the mobile phone, it reads the wallpaper from the local and displays it. It can be understood that when the mobile phone side detects a change in the local wallpaper, the mobile phone will perform the above operations to ensure that the background of the computer side and the security interface is consistent with the wallpaper of the mobile phone side.

[0183] It can be understood that Figure 6 This is only a schematic description and does not limit the embodiments of the present application. In some other embodiments, the security interface may further include other information. In addition, the information included in the security interface can be flexibly set. Details are not described herein.

[0184] The display stage of the security requirement interface:

[0185] After the mobile phone projects the first set of window interfaces to the laptop, if the user continues to operate on the mobile phone side or on the laptop side, it may trigger the display of an interface with security requirements. The following describes the embodiments of the display of the security interface from the perspectives of two scenarios: the user triggers the mobile phone and the user triggers the laptop.

[0186] Scenario 1: The user operates on the laptop side

[0187] In an embodiment of the present application, after the laptop receives an instruction for the user to click on the screen mirroring interface, the laptop can generate a first operation instruction, which is associated with a first APP. The first APP is the APP indicated by the image clicked by the user. Then, based on the WiFi Direct technology, the laptop sends the first operation instruction to the mobile phone. The mobile phone responds to the first operation instruction and performs an operation on the first APP, thereby triggering a change in the window interface displayed by the first APP. Further, the display of the mobile phone is updated. Further, the mobile phone performs a security flag bit detection and a re-screen mirroring operation on the updated display, so that the laptop displays the screen mirroring interface of the APP triggered by the user, realizing the function for the user to operate the APP on the mobile phone based on the screen mirroring interface. Figure 7 (a) to (b) in it illustrate an exemplary display method in Scenario 1. Figure 8 (a) to (b) in it illustrate along with Figure 7 the illustrated implementation process, the schematic change process of the display on the mobile phone side.

[0188] Such as Figure 7 as shown in (a) of it, GUI70 is the display interface of the laptop after the mobile phone mirrors the display80 shown in (a) of Figure 8 it to the laptop. GUI70 includes a screen mirroring interface 701 and a screen mirroring interface 702. The screen mirroring interface 701 is, for example, the screen mirroring interface of the mobile phone desktop, including images of APPs such as the mailbox and the memo. The screen mirroring interface 702 is, for example, the screen mirroring interface of the window interface of the memo. Among them, the screen mirroring interface 701 corresponds to the identifier task 0, and the screen mirroring interface 702 corresponds to the identifier task 1. After receiving an instruction for the user to click on the mailbox, the laptop generates a first operation instruction, which is used to trigger the mobile phone to run the mailbox APP (an instance of the aforementioned first APP). Then, the laptop sends the first operation instruction to the mobile phone. The mobile phone responds to the first operation instruction and runs the mailbox APP, thereby starting the login interface of the mailbox. The login interface of the mailbox is as shown in Figure 12 it, which will not be elaborated here. Further, the display80 on the mobile phone side is updated to Figure 8 display81 shown in (b) of it. Then, the mobile phone mirrors the display81 to the laptop, so that the GUI70 of the laptop is updated to Figure 7 GUI71 shown in (b) of it.

[0189] Among them, after the laptop receives the user's click operation, it can identify the first APP identifier corresponding to the image clicked by the user, and the first APP identifier is used to indicate the first APP. After that, the laptop generates a first operation instruction, and the first APP identifier is included in the first operation instruction. Furthermore, the mobile phone can determine the first APP according to the first APP identifier. The first APP identifier can be, for example, the name of the first APP.

[0190] Figure 7 The GUI71 shown in (b) in Figure 8 is the display interface of the laptop after the mobile phone mirrors the display81 shown in (b) in Figure 6 to the laptop. The GUI71 includes a mirroring interface 711, a mirroring interface 712, and a mirroring interface 713. The mirroring interface 711 corresponds to the identifier task 0, the mirroring interface 712 corresponds to the identifier task 1, and the mirroring interface 713 corresponds to the identifier task 2. Among them, the mirroring interface 711 is, for example, the mirroring interface of the mobile phone desktop, and the mirroring interface 711 is the same as the mirroring interface 701 in the GUI70. The mirroring interface 712 is, for example, the mirroring interface of the window interface of the memo, and the mirroring interface 712 is the same as the mirroring interface 702 in the GUI70. The mirroring interface 713 is, for example, the mirroring interface of the window interface of the mailbox, and the mirroring interface 713 is the security interface as shown in

[0191] As shown in (a) in Figure 8 , the display80 includes a window interface 71 and a window interface 72. The window interface 71 is the mobile phone desktop, and the mirroring results in the mirroring interface 701 in the GUI70. The window interface 72 is the window interface of the memo, and the mirroring results in the mirroring interface 702 in the GUI70. Due to the display of the window interface 71 and the window interface 72, the WMS extends a virtual display in units of the width of the mobile phone display screen, so that the display80 includes a display801 and a display802. The display801 is the display area corresponding to the display screen of the mobile phone, and the display802 is the virtual display. The width of the display802 can be the same as the width of the display801. Furthermore, the WMS deploys the window interface 71 to be displayed within the display801 and deploys the window interface 72 to be displayed within the display802. In addition, in order to facilitate the identification of the window interface 71 and the window interface 72, the AMS sets the identifier task 0 for the window interface 71 and sets the identifier task 1 for the window interface 72. Furthermore, in this embodiment, after the mobile phone determines that neither the window interface 71 nor the window interface 72 has security requirements, it sends task 0, task 1, and the video streams of the window interface 71 and the window interface 72 to the laptop, so that the laptop presents the GUI70.

[0192] Further, after the mobile phone responds to the first operation instruction and opens the login interface of the display email, a window interface is added to the display of the mobile phone on the basis of display80. Furthermore, display80 is updated to display81.

[0193] As Figure 8 shown in (b) of [], display81 is a schematic display interface obtained by screen mirroring GUI71. Display81 includes window interface 71, window interface 72, and window interface 73. Window interface 71 is screen-mirrored to obtain the screen-mirroring interface 711 in GUI71. Window interface 72 is screen-mirrored to obtain the screen-mirroring interface 712 in GUI71. Window interface 73 is screen-mirrored to obtain the screen-mirroring interface 713 in GUI71. In this embodiment, in order to accommodate window interface 71, window interface 72, and window interface 73, DMS extends a virtual display based on the display screen of the mobile phone to obtain display81. Display81 includes display811 and display812. Display811 is the display screen of the mobile phone, and display812 is the virtual display. The width of display812 can be twice the width of display811. Furthermore, WMS deploys window interface 71 to be displayed within display811, sets the identifier task 0 for window interface 71, and deploys window interface 72 and window interface 73 to be tiled and displayed within display812, sets the identifier task 1 for window interface 72, and sets the identifier task 2 for window interface 73.

[0194] In some embodiments, display81 can be obtained by the mobile phone continuing to operate on display80.

[0195] Furthermore, the WMS of the mobile phone traverses the Activities of window interface 71, window interface 72, and window interface 73 to determine whether each Activity contains a flag (which can be understood as determining whether the flag is set). Exemplarily, the WMS determines that neither the Activity of window interface 71 nor the Activity of window interface 72 contains a flag, while the Activity of window interface 73 contains a first flag. Subsequently, the mobile phone sends task 0, task 1, task 2, the video streams of window interface 71 and window interface 72, and a security protection instruction to the laptop. Among them, the video stream of window interface 71 corresponds to task 0, the video stream of window interface 72 corresponds to task 1, and the security protection instruction corresponds to task 2. Furthermore, the laptop splits the video streams of window interface 71 and window interface 72 and displays the screen mirroring interfaces 711 and 712. Moreover, the laptop responds to the security protection instruction and displays the screen mirroring interface of window interface 73 as a secure interface to obtain the screen mirroring interface 713.

[0196] In some embodiments, after the mobile phone mirrors a first group of window interfaces to the laptop, the display screen of the mobile phone can always display the mobile phone desktop. The window interfaces displayed in response to the operation instructions from the laptop can be displayed on a virtual display.

[0197] Scenario 2: The user operates on the mobile phone side

[0198] In the embodiments of the present application, after the mobile phone receives the second operation instruction input by the user, usually the second APP interface triggered by the instruction is displayed on the desktop of the mobile phone. In this way, the desktop of the mobile phone is updated, and then the display of the mobile phone is also updated accordingly. Then, the mobile phone can mirror the updated display to the laptop, so that the screen mirroring interface of the laptop is updated accordingly. Figure 9 (a) to (c) therein illustrate an exemplary display method in Scenario 2. Figure 10 (a) to (b) therein illustrate along with Figure 9 The schematic illustration shows the schematic change process of the display on the mobile phone side during the implementation process.

[0199] As Figure 9 As shown in (a) therein, GUI90 is the mobile phone that will Figure 10 After the display 101 shown in (a) is cast to the laptop, the display interface of the laptop. The GUI 90 includes a screen mirroring interface 901 and a screen mirroring interface 902. The screen mirroring interface 901 is, for example, the screen mirroring interface of the mobile phone desktop. The screen mirroring interface 902 is, for example, the screen mirroring interface of the window interface of the memo. Among them, the screen mirroring interface 701 corresponds to the identifier task 0, and the screen mirroring interface 701 corresponds to the identifier task 1.

[0200] As Figure 9 shown in (b) of, the GUI 91 is the desktop of the mobile phone, and the GUI 91 includes interface elements such as an email box. After the display 101 is cast to the laptop, the mobile phone, for example, receives a second operation instruction clicked and input by the user. The second operation instruction is, for example, used to start a second APP. The second APP is, for example, an email APP. Furthermore, since the user needs to operate the email on the mobile phone side, in this embodiment, the mobile phone displays the login interface of the email on the display screen. The login interface of the email is as Figure 12 shown, which will not be elaborated here. Correspondingly, the window interface corresponding to the mobile phone desktop is updated to the login interface of the email, and the display 101 is updated to Figure 10 the display 102 shown in (a) of. After that, the mobile phone casts the display 102 to the laptop, so that the GUI 91 of the laptop is updated to Figure 9 the GUI 92 shown in (c) of.

[0201] Figure 9 The GUI 92 shown in (c) of is the display interface of the laptop after the mobile phone casts the Figure 10 display 102 shown in (b) of to the laptop. The GUI 92 includes a screen mirroring interface 921 and a screen mirroring interface 922. The screen mirroring interface 921 is the screen mirroring interface of the window interface of the email, for example, the security interface as Figure 6 shown. The screen mirroring interface 922 is, for example, the screen mirroring interface of the window interface of the memo. Among them, the screen mirroring interface 921 corresponds to the identifier task 0, and the screen mirroring interface 922 corresponds to the identifier task 1.

[0202] As Figure 10 shown in (a) of, the display 101 includes a window interface 1011 and a window interface 1012. The DMS sets the identifier task 0 for the window interface 1011 and sets the identifier task 1 for the window interface 1012. The window interface 1011 is screen-mirrored to obtain the screen mirroring interface 901 in the GUI 90. The window interface 1012 is screen-mirrored to obtain the screen mirroring interface 90 in the GUI 90. The display 101 is similar to the Figure 8 display 80 shown in (a) of, which will not be elaborated here.

[0203] As shown Figure 10 in (b) of [reference], display 102 includes window interface 1021, window interface 1022, and window interface 1023. Among them, window interface 1021 is the same as window desktop 1011 and is the desktop of the mobile phone. Window interface 1022 is the same as window interface 1012 and is the interface of the memo. Window interface 1023 is the login interface of the mailbox, corresponding to the identifier task 2. That is, after the mobile phone responds to the second operation instruction and displays the login interface of the mailbox, AMS starts the mailbox APP, enters the mailbox login interface, and configures the label task 2 for the mailbox APP. DMS performs an extension based on display 101 to obtain display 102 so as to accommodate window interface 1023. Furthermore, in this embodiment, the mobile phone can project window interface 1022 and window interface 1023 onto the laptop. Specifically, when implementing, the mobile phone can determine through WMS that window interface 1023 is an interface with security requirements, while window interface 1022 has no security requirements. Then, the mobile phone sends task 1, task 2, the security protection instruction, and the video stream of window interface 1012 to the laptop. Among them, task 2 corresponds to the security protection instruction, and task 1 corresponds to the video stream of window interface 1012. Furthermore, the laptop displays GUI 92.

[0204] Based on the same principle as the embodiment Figure 9 and Figure 10 illustrated, in the Figure 5C-2 corresponding embodiment, when the mobile phone receives an instruction to trigger the display of window interface 561, or the instruction to trigger the display of window interface 561, the desktop interface of the mobile phone is updated to a window interface with security requirements. Furthermore, based on the updated window interface, the mobile phone projects the screen again, so that the projected screen corresponding to task 0 on the laptop side is a secure screen.

[0205] In other embodiments, if the interface displayed by the mobile phone in response to the second operation instruction has no security requirements, then the display screen of the mobile phone displays a window interface without security requirements. Furthermore, the mobile phone can project the image of the corresponding window interface onto the laptop. Correspondingly, the laptop can display the projected screen of the corresponding window interface and the projected screen of the memo window interface.

[0206] It can be seen that by adopting this implementation method, the mobile phone associates the window interface to be projected with the task ID. After the window interface is updated, the mobile phone can indicate the laptop to display the projected screen of each window interface through the task ID, so that the laptop can display the projected screen in units of window interfaces, which can not only securely display the interface with security requirements, but also ensure that the user can normally operate the interface without security requirements.

[0207] Display stage after operating the security interface:

[0208] Corresponding to the above Scenario 1, after the laptop displays the screen mirroring interface of the first APP as the security interface, the user can trigger the corresponding window interface to be displayed on the display screen of the mobile phone through this laptop to continue operating the first APP on the mobile phone side. Among them, the display of the window interface of the first APP on the display screen of the mobile phone and the user's continued operation will both trigger the update of the display on the mobile phone side, and further trigger the change of the screen mirroring interface displayed on the laptop side.

[0209] The following combines Figure 11 and Figure 12 , and describes the display method of the laptop and the change process of the display on the mobile phone side in the scenario where the window interface of the first APP is displayed on the display screen of the mobile phone.

[0210] Continuing from the aforementioned Scenario 1, Figure 11 the GUI of the laptop in (a) of Figure 6 is, for example, GUI71. Among them, the screen mirroring interface 713 includes an icon 1101, and the icon 1101 is the same as the icon 62 in Figure 12 . This is not elaborated here. After receiving the operation of the user clicking the icon 1101, the laptop generates a third operation instruction, and this third operation instruction is used to trigger the mobile phone to display the Figure 12 schematic email login interface on the display screen so that the user can perform a login operation on the email on the mobile phone side. Further, the window interface 73 in the display 81 on the mobile phone side is displayed on the display screen of the mobile phone, making the mobile phone present the Figure 11 interface. Further, the mobile phone can, based on the display 81, project the window interface 72 and the window interface 73 onto the laptop again, making the GUI71 of the laptop updated to the

[0211] schematic GUI111 in (b) of

[0212] Figure 11 The schematic GUI111 in (b) of Figure 8 After the display 81 shown in (b) is mirrored to the laptop, the display interface of the laptop. The GUI 111 includes a mirroring interface 1111 and a mirroring interface 1112. The mirroring interface 1111 corresponds to the identifier task 0, and the mirroring interface 1112 corresponds to the identifier task 1. Among them, the mirroring interface 1111 is the mirroring interface of the email login interface, and the mirroring interface 1111 is a security interface as shown in Figure 6 The mirroring interface 1112 is the mirroring interface of the memo window interface, and the mirroring interface 1112 is the same as the mirroring interface 712 in the GUI 71, which will not be elaborated here.

[0213] Combined with Figure 8 the display 81 shown in (b), in this embodiment, the mobile phone can obtain the video stream of the window interface 72 based on the starting coordinates in the upper left corner of the window interface 72 and render it. Then, it sends task 1, task 2, the rendered video stream, and the security protection instruction to the laptop, where task 1 corresponds to the video stream and task 2 corresponds to the security protection instruction. Furthermore, the laptop displays the mirroring interface 1112 corresponding to task 1 and the mirroring interface 1111 corresponding to task 2. It should be noted that in this embodiment, the mirroring information sent by the mobile phone does not include task 0 and the information corresponding to task 0 (i.e., the interface of the mobile phone desktop) in the display 81. Therefore, the GUI 111 does not include the mirroring interface corresponding to task 0. From the user's visual effect, when the GUI 71 is updated to the GUI 111, the laptop deletes the mirroring interface 713 and updates the mirroring interface 711 to a security interface.

[0214] Figure 12 The GUI 120 shown is the email login interface on the mobile phone side. The GUI 120 includes interface elements such as a control 1201 for entering the "email account", a control 1202 for entering the "password", and a "login" button 1203. Among them, the control 1202 includes a reminder message "Enter password", and the font gray value of the reminder message "Enter password" is relatively low to remind the user to enter the login password of the email in this control. After receiving the password entered by the user, the reminder message "Enter password" is hidden. In some embodiments, the control 1201 includes a reminder message "Enter email account", and the font gray value of the reminder message "Enter email account" is relatively low to remind the user to enter the login account of the email in this control. After receiving the account entered by the user, the reminder message "Enter email account" is hidden. In other embodiments, the control 1201 includes the login account of the email, and the control 1202 includes a reminder message "Enter password". In the scenario where the mobile phone displays the GUI 120 through the display screen, the display of the mobile phone is updated from display 81 to display 142.

[0215] Further, in combination with Figures 13 to 14B , after the mobile phone receives the operation performed by the user on the first APP, the display method of the laptop and the change process of the display on the mobile phone side will be described.

[0216] It should be noted that Figure 11 The implementation scenario corresponding to (b) in Figure 9 is the same as the implementation scenario corresponding to (c) in . Based on this, the following embodiments of the display method are applicable to both of the above two implementation scenarios, and the embodiments of the present application will not be described repeatedly.

[0217] In combination with Figure 12 , the mobile phone can receive the login instruction input by the user clicking the button 1203, and the login instruction includes the email account and password input by the user. After determining that the corresponding email account and password are correct, the AMS calls the Activity of the email main interface. Correspondingly, the mobile phone displays the main interface of the email. The main interface of the email is as shown in the window interface identified by task 2 in Figure 13 .

[0218] Figure 13 The schematic display 142 shown in Figure 13 includes a window interface 1421, a window interface 1422, and a window interface 1423. The window interface 1421 corresponds to the identification task 0, which is the mobile phone desktop. The window interface 1422 corresponds to task 1 and is the window interface of the memo. The window interface 1423 corresponds to task 2 and is the window interface of the email main interface. In this embodiment, after determining that both the email account and password are correct, the DMS can control the login interface corresponding to task 0 in the display 81 to be closed or hidden, and display the email main interface through a new window interface. In this way, the window interface identified by task 0 is updated to the mobile phone desktop, and a new window interface is added. Further, the AMS configures the identification task 2 for the new window interface to obtain the

[0219] schematic display 142 shown in Figure 14A In some embodiments, the mobile phone can project the window interface 1422 and the window interface 1423 in the display 142 to the laptop side to obtain the Figure 14B schematic GUI 151 shown in . In other embodiments, the mobile phone can project all the window interfaces in the display 142 to the laptop side to obtain the

[0220] As shown in Figure 14A As shown, GUI 151 includes a screen mirroring interface 1511 and a screen mirroring interface 1512. Among them, the screen mirroring interface 1511 corresponds to task 1 and is the screen mirroring interface of the window interface 1422 in display 142. The screen mirroring interface 1512 corresponds to task 2 and is the screen mirroring interface of the window interface 1423 in display 142.

[0221] As Figure 14B shown, GUI 152 includes a screen mirroring interface 1521, a screen mirroring interface 1522, and a screen mirroring interface 1523. Among them, the screen mirroring interface 1521 corresponds to task 0 and is the screen mirroring interface of the window interface 1421 in display 142. The screen mirroring interface 1522 corresponds to task 1 and is the screen mirroring interface of the window interface 1422 in display 142. The screen mirroring interface 1523 corresponds to task 2 and is the screen mirroring interface of the window interface 1423 in display 142.

[0222] It should be noted that the interaction of various information and instructions between the above-mentioned mobile phone and laptop can all be based on the WiFi Direct technology.

[0223] It can be understood that Figures 5A to 14B it is only a schematic description and does not limit the embodiments of the present application. In other embodiments, the embodiments of the above display method are equally applicable to the display of other mobile phone APPs, which will not be described one by one here. In addition, the GUI in the above scenarios may vary according to different mobile phone models, system versions, etc. Details are not described here.

[0224] In addition, the above embodiments are only described by taking a mobile phone and a laptop as examples, and do not limit the embodiments of the present application. In other embodiments, the first device involved in the embodiments of the present application can also be implemented as a foldable screen device, a tablet computer, etc., and the second device involved in the embodiments of the present application can also be implemented as a smart screen, etc. There is no limitation here.

[0225] The following takes the first device implemented as a mobile phone and the second device implemented as a PC as an example to describe the overall process of the display method involved in the embodiments of the present application.

[0226] See Figure 15 , Figure 15 Illustrates the third exemplary display method of the present application. In some embodiments, multi-screen collaboration is performed on the mobile phone side. The mobile phone side first determines whether it is currently in multi-screen collaboration. If not, the default application lock policy is adopted (that is, the mobile phone side will not send the video stream to the PC side). If it is, the mobile phone side determines whether it is in a secure input interface. If not, the default window display is used. If so, the task ID of the application corresponding to the secure input interface and the covering screen (the covering screen can also be optionally pre-stored on the PC) are sent to the PC. After the user finishes entering the password on the mobile phone (the flag corresponding to the window state is reset), the PC is notified. The PC cancels the covering of the application window corresponding to the secure input interface.

[0227] It can be understood that, compared with the traditional solution, when encountering the security flag bit flag, the content on the display will not be projected to the opposite device. The solution of the embodiment of the present application will project the content on the display to the opposite device regardless of whether there is a security flag bit flag. At the same time, the security flag bit flag is transmitted to tell the PC to perform a security layer covering on the existence of the security flag bit flag, realizing the security guarantee of multi-screen collaboration and multiple windows.

[0228] In summary, for the display method of the embodiment of the present application, the first device determines whether each window interface has a security requirement before screen mirroring. Furthermore, the first device sends the task ID of each window interface and the screen mirroring information of the window interface to the second device according to the detection result. The second device displays the secure interface and the screen mirroring interface containing the content of the window interface based on the task ID, where the secure interface is the screen mirroring interface of the window interface with security requirements. In this way, the second device can not only securely display the interface with security requirements but also ensure the normal operation of the user on the interface without security requirements.

[0229] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0230] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these changes and modifications.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A display method, characterized in that, the method includes: The first device receives a screen mirroring instruction input by the user; In response to the screen mirroring instruction, the first device screens the first window interface to the second device, so that the second device displays the first window interface, and the first window interface displays the first content; The first device runs a second window interface in response to an instruction, and the second window interface displays second content; When the first device determines that there is no security flag bit in the second window interface, the first device screens the second window interface to the second device, so that the second device simultaneously displays the first window interface and the second window interface; In response to a user operation, the second window interface of the first device displays third content, and the second content and the third content are different interfaces of a first App; When the first device determines that there is a security flag bit in the second window interface, the first device sends the task number corresponding to the second window interface and a security protection instruction to the second device, and the security protection instruction is used to instruct the second device to display the screen mirroring interface of the second window interface as a secure interface, the secure interface does not include the third content, and the first window interface still displays the first content.

2. The method according to claim 1, characterized in that, The first device runs a second window interface in response to an instruction, including: The first device receives a first operation instruction from the second device, the first operation instruction is associated with a first application APP, and the second window interface is a window interface of the first APP; The first device runs the first APP to start the second window interface.

3. The method according to claim 1, characterized in that, The first device runs a second window interface in response to an instruction, including: The first device receives a second operation instruction input by the user, the second operation instruction corresponds to a first application APP, and the second window interface is a window interface of the first APP; The first device runs the first APP and displays the second window interface.

4. The method according to claim 1 or 2, characterized in that, After the first device sends the task number corresponding to the second window interface and a security protection instruction to the second device, it further includes: The first device receives a third operation instruction from the second device, and the third operation instruction includes the task number corresponding to the second window interface; In response to the third operation instruction, the first device displays the second window interface; The first device receives operation information input by the user based on the second window interface; The first device updates the second window interface to a third window interface; The first device screens the third window interface to the second device.

5. The method according to any one of claims 1-4, characterized in that, The first device determines that there is a security flag bit in the second window interface, including: The first device calls the window management service of the first device to traverse the window state of the second window interface; If the window state of the second window interface includes a security flag, the first device determines that the second window interface includes a security flag bit.

6. The method according to claim 5, wherein, the security flag includes a first flag, a second flag, and a third flag. The first flag indicates an application security lock, the second flag indicates an application active setting, and the third flag indicates a call to a security keyboard.

7. The method according to claim 1, wherein, the security interface includes a first prompt message for prompting the user to perform an operation on the first device side.

8. An electronic device, wherein, serving as the first device, the electronic device includes a memory and one or more processors; wherein, the memory is used to store computer program code, and the computer program code includes computer instructions; when the computer instructions are executed by the processor, the electronic device executes the display method according to any one of claims 1 to 7.

9. A computer-readable storage medium, wherein, including a computer program, when the computer program runs on a computer, the computer executes the display method according to any one of claims 1 to 7.

10. A computer program product, wherein, including instructions, when the instructions run on a computer, the computer executes the display method according to any one of claims 1 to 7.

Images