Display method and electronic equipment

Abstract

The embodiment of the invention discloses a display method and electronic equipment. The first equipment disclosed by the embodiment of the invention can project the screen to the second equipment. Specifically, after the first device receives a screen projection instruction input by a user, a first window interface is projected to the second device. A first device receives an instruction for operating a second window interface, and after it is determined that the second window interface contains a security flag bit, a task number of the second window interface and a security protection instruction can be sent to a second device. And further, the second device displays the first window interface and a security interface, and the security interface is a screen projection interface of the second window interface. Therefore, the second equipment can safely display the interface with the safety requirement, and can also ensure that the user normally operates the interface without the safety requirement, so that the use experience of the user is improved.

Description

[0001] This application is a divisional application. The application number of the original application is 202010949251.X, and the original application date is September 10, 2020. The entire content of the original application is incorporated into this application by reference. Technical Field

[0002] The embodiments of the present application relate to the field of display, and in particular to a display method and electronic device. Background Art

[0003] Screen projection is a technology that transmits the window interface of device 1 to device 2 for display. For example, Figure 1 As shown, the mobile phone can transmit the window interface of the mobile phone to the computer for display. In this way, the computer can display the screen projection interface 01 of the mobile phone window interface on the screen. The user can operate the application (APP) of the mobile phone through the screen projection interface 01.

[0004] Apps running on mobile phones include those with security requirements, such as those that require users to enter a password. Therefore, when mirroring, the password entered on the phone will be displayed on the computer, including visible passwords, password length, and biometric passwords, posing a potential privacy risk. Summary of the Invention

[0005] The embodiments of the present application provide a display method and an electronic device, which can solve the problem of poor user experience caused by existing display methods.

[0006] In an embodiment of the present application, the first device can project multiple window interfaces of the first device to the second device.

[0007] In a first aspect, an embodiment of the present application provides a display method, the method comprising: a first device receives a screen projection instruction input by a user; in response to the screen projection instruction, the first device projects the first window interface to a second device; the first device responds to the instruction to run the second window interface; the first device determines that the second window interface contains a security flag; the first device sends a task number corresponding to the second window interface and a security protection instruction to the second device, the security protection instruction being used to instruct the second device to display the projection interface of the second window interface as a security interface.

[0008] The first device can configure the application display area (display) and the display strategy of the window interface running on the first device side within the display. Then, the first device can project the screen to the second device based on the display. In addition, the first device configures taskIDs for the apps running on the first device side to represent the window interfaces of each app.

[0009] In an embodiment of the present application, the first device determines whether each window interface has security requirements before screen projection. Then, based on the detection results, the first device sends the taskID of each window interface and the projection information of the window interface to the second device. The second device displays the security interface and the projection interface containing the window interface content based on the taskID, wherein the security interface is the projection interface of the window interface with security requirements. In this way, the second device can not only safely display the interface with security requirements, but also ensure that the user can operate normally on the interface without security requirements.

[0010] In one possible design, the first device responds to an instruction to launch a second window interface, including: the first device receives a first operation instruction from the second device, the first operation instruction is associated with a first application program (APP), and the second window interface is a window interface of the first APP; the first device launches the first APP to launch the second window interface. The first device can communicate with the second device based on WiFi Direct technology. This enables the user to operate the APP of the first device on the second device.

[0011] In one possible design, the first device responds to an instruction to run a second window interface, including: the first device receives a second operation instruction input by a user, the second operation instruction corresponds to a first application APP, and the second window interface is a window interface of the first APP; the first device runs the first APP and displays the second window interface.

[0012] In one possible design, after the first device sends the task number and security protection instruction corresponding to the second window interface to the second device, it also includes: the first device receives a third operation instruction from the second device, and the third operation instruction includes the task number corresponding to the second window interface; in response to the third operation instruction, the first device displays the second window interface; the first device receives the operation information input by the user based on the second window interface; the first device updates the second window interface to the third window interface; the first device projects the third window interface to the second device. After the second device displays the projection interface of the second window interface as a security interface, the user can trigger the second window interface to be displayed on the display screen of the first device through the second device to continue operating the second window on the first device side. The first device can determine the second window interface by the task number. Using this implementation method, the first device associates the window interface with the task ID, so that both the first device and the second device can determine the window interface by the task ID, making it easy for the user to switch the window interface displayed on the first device side and the second device side, thereby improving the user experience.

[0013] In one possible design, the first device determines that the second window interface contains a security flag, including: the first device calls the window management service of the first device to traverse the window status of the second window interface; if the window status of the second window interface contains a security flag, the first device determines that the second window interface contains a security flag.

[0014] In one possible design, the security flag includes a first flag, a second flag, and a third flag. The first flag indicates that a security lock is applied, the second flag indicates that the application is actively set, and the third flag indicates that a security keyboard is called.

[0015] In the second aspect, an embodiment of the present application provides a display method, which includes: a second device receives a video stream from a first window interface of a first device; the second device displays an image corresponding to the video stream to obtain a projection interface of the first window interface; the second device receives a task number and security protection instructions from the first device; the second device displays the projection interface and security interface of the first window interface.

[0016] In one possible design, after the second device displays the projection interface and security interface of the first window interface, it also includes: the second device receives a click instruction input by the user, and the click instruction corresponds to the security interface; the second device generates an operation instruction, and the operation instruction includes a task number; the second device sends the operation instruction to the first device.

[0017] In a third aspect, an embodiment of the present application provides an electronic device having the function of implementing the first device in the above method. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the structure of the above electronic device includes a processor and a transceiver, and the processor is configured to process the electronic device to execute the corresponding function of the first device in the above method. The transceiver is used to implement information exchange with a second device. The electronic device may also include a memory, which is used to couple with the processor and store program instructions and data necessary for the electronic device.

[0018] In a fourth aspect, an embodiment of the present application provides an electronic device having the function of implementing the second device in the above method. The function can be implemented by hardware, or it can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the structure of the above electronic device includes a processor, a receiver and a display, and the processor is configured to process the electronic device to perform the corresponding functions in the above method. The transceiver is used to implement information interaction with the first device. The display is used to realize the display of the projection interface and the security interface. The electronic device may also include a memory, which is used to couple with the processor and store the necessary program instructions and data for the electronic device.

[0019] In a fifth aspect, an embodiment of the present application provides a computer storage medium, which stores instructions. When the instructions are run on a computer, the computer executes part or all of the steps of the display method in the first aspect, the second aspect, various possible implementations of the first aspect, and various possible implementations of the second aspect.

[0020] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when running on a computer, enables the computer to execute part or all of the steps of the display method in the first aspect, the second aspect, various possible implementations of the first aspect, and various possible implementations of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 A schematic diagram of a typical screen projection scenario provided in an embodiment of the present application;

[0023] Figure 2A This is a schematic diagram of an exemplary hardware structure of the first device 100 provided in an embodiment of the present application;

[0024] Figure 2B A schematic diagram of an exemplary software architecture of the first device 100 provided in an embodiment of the present application;

[0025] Figure 2C A schematic diagram of an exemplary architecture of a second device 200 provided in an embodiment of the present application;

[0026] Figure 3 An exemplary plan view of an application display area (display) provided in an embodiment of the present application;

[0027] Figure 4A A schematic diagram of a first exemplary signaling interaction of the display method provided in an embodiment of the present application;

[0028] Figure 4B A schematic diagram of a second exemplary signaling interaction of the display method provided in an embodiment of the present application;

[0029] Figure 5A A schematic diagram of a first exemplary interface for a screen projection scenario provided in an embodiment of the present application;

[0030] Figure 5B A second exemplary interface diagram of a screen projection scenario provided in an embodiment of the present application;

[0031] Figure 5C A third exemplary interface diagram of the screen projection scenario provided in an embodiment of the present application;

[0032] Figure 5C-1 Provided in the embodiments of this application Figure 5C An exemplary schematic diagram of the display corresponding to the illustrated scenario;

[0033] Figure 5C-2 Provided in the embodiments of this application Figure 5C A schematic diagram of an exemplary interface of a mobile phone in the illustrated scenario;

[0034] Figure 6 An exemplary interface diagram of a security interface provided in an embodiment of the present application;

[0035] Figure 7 This is a schematic diagram of an exemplary interface of the first group of screen projection interfaces provided in an embodiment of the present application;

[0036] Figure 8 Provided in the embodiments of this application Figure 7 An exemplary schematic diagram of a display corresponding to the display interface;

[0037] Figure 9This is a schematic diagram of an exemplary interface of the second group of screen projection interfaces provided in an embodiment of the present application;

[0038] Figure 10 Provided in the embodiments of this application Figure 9 An exemplary schematic diagram of a display corresponding to the display interface;

[0039] Figure 11 This is an exemplary interface diagram of the third group of screen projection interfaces provided in an embodiment of the present application;

[0040] Figure 12 This is a schematic diagram of an exemplary interface of the mailbox login interface provided in an embodiment of the present application;

[0041] Figure 13 An exemplary schematic diagram of a display provided in an embodiment of the present application;

[0042] Figure 14A Provided in the embodiments of this application Figure 13 Schematic diagram of a first exemplary screen projection interface of a display;

[0043] Figure 14B Provided in the embodiments of this application Figure 13 Schematic diagram of a second exemplary screen projection interface of a display;

[0044] Figure 15 A third exemplary signaling interaction diagram of the display method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions of the embodiments of the present application will be clearly described below in conjunction with the drawings in the embodiments of the present application.

[0046] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear contrary indication in the context. It should also be understood that although the terms first, second, etc. may be used to describe a certain class of objects in the following embodiments, the objects should not be limited to these terms. These terms are only used to distinguish the specific objects of this class of objects. For example, the terms first, second, etc. may be used to describe devices in the following embodiments, but devices should not be limited to these terms. These terms are only used to distinguish different electronic devices. The same applies to other classes of objects that may be described by the terms first, second, etc. in the following embodiments, which will not be repeated here.

[0047] An embodiment of the present application provides a display method, which is applied to a scenario in which a first device projects at least two window interfaces onto a second device. For a window interface without security requirements, the second device displays the content of that window interface, and for a window interface with security requirements, the second device displays the security interface. This allows the second device to both safely display the interface with security requirements and ensure normal user operation of the interface without security requirements.

[0048] The following introduces a first device, a second device, a user interface (UI) for such a first device and such a second device, and embodiments for using such a first device and such a second device.

[0049] To clearly describe the embodiments of this application, the interface displayed by the first device is referred to as a "window interface" in this specification. A "window interface" is a display interface whose size is the same as the display screen of the first device. The interface displayed by the second device is referred to as a "screen projection interface" in this specification. Each "screen projection interface" corresponds to a "window interface."

[0050] The first device involved in the embodiment of the present application may be an electronic device including a wireless communication module and a display function, such as a mobile phone, a tablet computer, a wearable device (for example, a watch, a bracelet, a helmet, etc.), a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a smart home device (for example, a smart TV, etc.), and other electronic devices. It is understandable that the embodiment of the present application does not impose any restrictions on the specific type of the first device. The first device may be a device equipped with Or devices with other operating systems.

[0051] Figure 2A: The figure shows a hardware structure diagram of the first device 100. The first device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0052] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the first device 100. In other embodiments of the present application, the first device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0053] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Among them, different processing units can be independent devices or integrated into one or more processors. In some embodiments, the first device 100 may also include one or more processors 110.

[0054] The controller may be the nerve center and command center of the first device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of the detection instruction and the like.

[0055] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the first device 100.

[0056] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0057] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C bus lines. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via the I2C interface, allowing the processor 110 and the touch sensor 180K to communicate via the I2C bus interface, thereby implementing the touch function of the first device 100.

[0058] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface, enabling the function of listening to audio through a Bluetooth headset.

[0059] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of listening to audio through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0060] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface, enabling the function of playing music through Bluetooth headphones.

[0061] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the first device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the first device 100.

[0062] The GPIO interface can be configured via software. The GPIO interface can be configured as either a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0063] The USB interface 130 is an interface that complies with USB standards and specifications, and may be a MiniUSB interface, a MicroUSB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the first device 100, or to transfer data between the first device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as AR devices.

[0064] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely illustrative and does not constitute a structural limitation on the first device 100. In other embodiments, the first device 100 may also adopt a different interface connection method from the above embodiments, or a combination of multiple interface connection methods.

[0065] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the first device 100. While charging the battery 142, the charging management module 140 can also provide power to the first device 100 via the power management module 141.

[0066] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0067] The wireless communication function of the first device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0068] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in first device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0069] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the first device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0070] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays the operating interface of the APP through the display 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0071] The wireless communication module 160 can provide wireless communication solutions applied on the first device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0072] In some embodiments, the antenna 1 of the first device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the first device 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0073] In some embodiments, the wireless communication solution provided by the mobile communication module 150 can enable the first device 100 to communicate with devices in the network (such as a cloud server). The WLAN wireless communication solution provided by the wireless communication module 160 can also enable the first device 100 to communicate with devices in the network (such as the second device 200), so that the first device 100 can transmit a video stream to the second device 200, thereby realizing the function of casting the screen of the first device 100 to the second device 200.

[0074] The first device 100 implements a display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. For example, the GPU can render an image as a security interface. The processor 110 may include one or more GPUs, which execute program instructions to generate or change the interface display effect. In an embodiment of the present application, the display screen 194 may include a display and a touch device. The display is used to output display content to the user, such as the desktop page of the first device 100, and various operation interfaces of the APP. The touch device is used to receive operation instructions input by the user on the display screen 194.

[0075] The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, a quantum dot light-emitting diode (QLED), or the like.

[0076] In some embodiments of the present application, when the display panel uses materials such as OLED, AMOLED, or FLED, the display screen 194 can be bent. Here, the display screen 194 being bendable means that the display screen can be bent at any position to any angle and can be maintained at that angle. For example, the display screen 194 can be folded in half from the middle, left to right, or vertically from the middle.

[0077] The first device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0078] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0079] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB, YUV, etc.

[0080] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the first device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0081] A video codec is used to compress or decompress digital video. In some embodiments, after the first device 100 is connected to the second device 200, the video codec can compress the video stream to be projected to the second device 200 and then send the compressed video file to the second device 200. The first device 100 can support one or more video codecs.

[0082] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications of the first device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0083] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the first device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement data storage functions. For example, data such as music, photos, and videos can be stored in the external memory card.

[0084] The internal memory 121 can be used to store one or more computer programs, which include instructions. The processor 110 can run the above instructions stored in the internal memory 121, so that the first device 100 executes the display method provided in some embodiments of the present application, as well as various functional applications and data processing. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as mailboxes, galleries, contacts, etc.). The data storage area can store data created during the use of the first device 100 (such as window interface logos, etc.). In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0085] The first device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0086] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0087] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The first device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0088] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the first device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0089] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The first device 100 can be provided with at least one microphone 170C. In other embodiments, the first device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the first device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, and directional recording functions, etc.

[0090] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0091] The pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The first device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the first device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The first device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0092] The gyroscope sensor 180B can be used to determine the motion posture of the first device 100. In some embodiments, the angular velocity of the first device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the first device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the first device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0093] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the first device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

[0094] The magnetic sensor 180D includes a Hall sensor. The first device 100 can use the magnetic sensor 180D to detect the orientation of the display screen 194 of the first device 100. In some embodiments, when the first device 100 is a flexible screen device, the first device 100 can detect the folded and unfolded state of the screen based on the magnetic sensor 180D.

[0095] The accelerometer 180E can detect the magnitude of the acceleration of the first device 100 in all directions (generally three axes). When the first device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the first device 100, which is applicable to applications such as switching between landscape and portrait modes and pedometers.

[0096] The distance sensor 180F is used to measure distance. The first device 100 can measure distance using infrared or laser. In some embodiments, when shooting a scene, the first device 100 can use the distance sensor 180F to measure distance to achieve fast focus.

[0097] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The first device 100 emits infrared light outward through the light emitting diode. The first device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the first device 100. When insufficient reflected light is detected, the first device 100 can determine that there is no object near the first device 100. The first device 100 can use the proximity light sensor 180G to detect that the user is holding the first device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0098] The ambient light sensor 180L is used to sense ambient light brightness. The first device 100 can adaptively adjust the brightness of the display screen 194 based on the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking photos. The ambient light sensor 180L can also work with the proximity light sensor 180G to detect whether the first device 100 is in a pocket to prevent accidental touches.

[0099] The fingerprint sensor 180H is used to collect fingerprints. The first device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0100] The temperature sensor 180J is used to detect temperature. In some embodiments, the first device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the first device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the first device 100 heats the battery 142 to prevent the first device 100 from shutting down abnormally due to low temperature. In other embodiments, when the temperature is lower than another threshold, the first device 100 boosts the output voltage of the battery 142 to prevent abnormal shutdown due to low temperature.

[0101] The touch sensor 180K may also be referred to as a touch panel or touch-sensitive surface. The touch sensor 180K may be provided on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also referred to as a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor may transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations may be provided through the display screen 194. In other embodiments, the touch sensor 180K may also be provided on the surface of the first device 100, at a location different from that of the display screen 194.

[0102] The bone conduction sensor 180M can obtain vibration signals. In some embodiments, the bone conduction sensor 180M can obtain vibration signals from the vibrating bones of the human body. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure pulse signals. In some embodiments, the bone conduction sensor 180M can also be set in headphones to form bone conduction headphones. The audio module 170 can parse out voice signals based on the vibration signals of the vibrating bones of the human body obtained by the bone conduction sensor 180M to implement voice functions. The application processor can parse heart rate information based on the blood pressure pulse signals obtained by the bone conduction sensor 180M to implement heart rate detection functions.

[0103] The buttons 190 include a power button, a volume button, etc. The buttons 190 may be mechanical buttons or touch buttons. The first device 100 may receive button inputs and generate key signal inputs related to user settings and function control of the first device 100.

[0104] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0105] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0106] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and disconnected from the first device 100 by inserting it into or removing it from the SIM card interface 195. The first device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The first device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the first device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the first device 100 and cannot be separated from the first device 100.

[0107] The software system of the first device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture. The embodiment of the present application takes the Android system with a layered architecture as an example to exemplify the software structure of the first device 100.

[0108] Figure 2B It is a schematic diagram of the software structure of the first device 100 according to an embodiment of the present application.

[0109] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system consists of, from top to bottom, the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0110] The application layer can include a series of application packages.

[0111] like Figure 2BAs shown, the application package may include applications such as gallery, calendar, map, WLAN, music, SMS, call, navigation, Bluetooth, video, etc.

[0112] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0113] like Figure 2B As shown, the application framework layer may include a display management service (DMS), an activity management service (AMS), and a window management service (WMS). The window management service may also be called a window manager. Of course, the application framework layer may also include a content provider, a view system, a phone manager, a resource manager, a notification manager, etc., but this embodiment of the application does not impose any restrictions on this.

[0114] Among them, WMS can be used to determine the display strategy of the APP in the first device 100. Afterwards, WMS can control the APP to be displayed on the display screen 194 of the first device 100 according to the display strategy through the display driver and surface manager of the underlying display system.

[0115] Among them, DMS can be used to configure the display strategy of the window interface to be projected to the second device 200 in the application display area (display) of the first device 100, so that the second device 200 can display the corresponding projection interface based on the display strategy of the window interface in the display.

[0116] Among them, display is the display area of ​​the application running on the first device, which can also be understood from the perspective of screen projection as the presentation area configured by WMS for the window interface to be projected. WMS can configure display according to the size of the area occupied by the window interface to be projected when it is tiled. Figure 3 As shown, WMS can use the upper left corner of the first device 100 as the origin of the coordinate system and the width of the display screen 194 of the first device 100 as the unit to configure the aforementioned display. For example, WMS can expand in the x-axis direction to obtain the aforementioned display. The size of the display is associated with the number of window interfaces to be projected. For example, Figure 3In the embodiment shown in (a), the window interface to be projected is a scene, the display area of ​​the display screen 194 can accommodate the window interface to be projected, and the WMS configures the display area of ​​the display screen 194 to display30. For another example, Figure 3 In the embodiment shown in (b), in a scenario where there are two window interfaces to be projected, the display area of ​​the display screen 194 cannot accommodate the two window interfaces displayed in a tiled manner. Based on this, the WMS expands a display area of ​​the width of the display screen 194 in the x-axis direction. Then, the WMS configures one window interface to be projected to be displayed in the display area of ​​the display screen 194, and the other window interface to be projected to be displayed in the expanded area. That is, Figure 3 In the embodiment shown in (b), display31 includes a first display311 and a second display312. The first display311 is configured on the display screen 194 of the first device 100 and is a display visible to the user. The second display312 is an extended display of the first display311 and is a display invisible to the user. The embodiment of the present application also refers to the extended display as a "virtual display". It should be understood that the size of the virtual display is associated with the number of window interfaces to be projected. For example, in a scenario where there are three window interfaces to be projected, WMS expands a display area with a width of two display screens 194 in the x-axis direction. Correspondingly, the width of the virtual display is the width of two display screens 194.

[0117] AMS can be used to manage the running processes of apps, including starting, pausing, and terminating them. An app typically consists of multiple activities, and AMS is specifically used to manage different activities within different app processes. For example, when the login interface of the email app is running, AMS calls an activity corresponding to the email app. During the process of displaying the main email interface, AMS calls another activity corresponding to the email app.

[0118] AMS can also be used to set a task number (taskidentity, taskID) for the started APP after an APP process is started, so that all activities of the APP correspond to the set taskID, so that the first device 100 can distinguish between each window interface, and the second device 200 can distinguish between each projection interface. Exemplarily, AMS can set taskID for APP according to the number of APPs running on the first device 100. For example, AMS can set the task number of the main interface of the first device 100 to task0. After the process of the mailbox APP in the first device 100 is started, AMS sets the task number "task1" for the mailbox APP. In other embodiments, AMS can also be used to delete the task number corresponding to the corresponding APP after an APP process is terminated.

[0119] WMS can be used to manage window programs. For example, WMS can obtain the size of the display screen, the size of the display, and the position and coordinates of each window interface. In addition, WMS can also traverse whether the current interface of the running APP contains a security flag (by checking the parameters of windowstate). After traversing to the current interface (Activity) of the APP that contains a security flag, the first device 100 is triggered to send a security protection instruction to the second device 200, so that the second device 200 displays the security interface at the forefront of the projection interface of the APP window interface, and at the same time terminates the projection of the display where the application is located to the second device 200. The security protection instruction, for example, includes the task number of the corresponding window interface.

[0120] In some embodiments, the security flag can be implemented as a security flag, for example. In actual implementation, the security flag can include multiple types of flags, and different types of flags can indicate security requirement scenarios of different natures. For example, the first flag can indicate an application security lock. In actual operation, when an application with a security lock receives a user operation to open it, it will trigger the security mode of the application security lock. The second flag can be set actively by the application, for example. In some embodiments, the implementation scenarios of biometrics (such as face unlocking) or display interfaces containing private information (such as ID card photos) trigger the security mode set actively by the application. In other embodiments, some applications set that the current activity cannot take screenshots or record the screen, triggering the security mode set actively by the application. The third flag can indicate the call of a security keyboard. In actual operation, the implementation scenario of entering a bank card account and / or password triggers the call of the security mode of the security keyboard. It can be understood that in some embodiments, the flag is associated with the activity, that is, one activity corresponds to one flag.

[0121] In some embodiments, the presence or absence of a flag referred to in this application can be understood as determining the value of the flag. For example, if the flag is 1, it is determined to be present, and if the flag is 0, it is determined not to be present. For another example, if the flag is 0, it is determined to be present, and if the flag is 1, it is determined not to be present. This application does not limit this.

[0122] Based on the description of the aforementioned embodiment, the window interface can correspond to multiple Activities. During the process of calling the Activity of the window interface, the Windowstate of the corresponding APP is updated. In some embodiments, during the implementation of updating the Windowstate, it can be updated to at least one of the aforementioned flags. Based on this, the WMS can determine whether the window interface contains a security flag by traversing whether the Windowstate of the window interface contains a flag. If the Windowstate corresponding to the corresponding Activity contains any of the aforementioned flags, it indicates that the current window interface of the APP contains a security flag, and the first device 100 sends a security protection instruction to the second device 200.

[0123] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0124] The view system includes visual controls, such as those that display text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, consider a display interface that includes a camera icon.

[0125] The phone manager is used to provide communication functions of the first device 100, such as management of call status (including answering, hanging up, etc.).

[0126] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0127] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short period of time without user interaction. For example, the notification manager is used to notify the completion of downloads, message reminders, etc. The notification manager can also be used to display notifications in the form of icons or scrolling text in the top status bar of the system, such as notifications from applications running in the background, or notifications that appear on the screen in the form of dialog windows. For example, a text message can be displayed in the status bar, a prompt sound can be emitted, the first device 100 can vibrate, an indicator light can flash, etc.

[0128] The Android Runtime consists of the core library and the virtual machine. The Android runtime is responsible for scheduling and management of the Android system.

[0129] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0130] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0131] Continue to see Figure 2B The system libraries and kernel layers below the application framework layer can be called the underlying system. The underlying system includes the underlying display system for providing display services. For example, the underlying display system includes the display driver in the kernel layer and the surface manager in the system library.

[0132] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0133] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0134] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0135] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0136] A 2D graphics engine is a drawing engine for 2D drawings.

[0137] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.

[0138] The second device involved in the embodiment of the present application may be an electronic device including a wireless communication module and a display function, such as a display, a smart TV, a tablet computer, a laptop computer, a UMPC, a netbook, and other electronic devices. It is understandable that the embodiment of the present application does not impose any restrictions on the specific type of the second device. The second device may be a device equipped with Or devices with other operating systems.

[0139] Figure 2C FIG2 shows an exemplary architecture diagram of the second device 200. The second device 200 may include a processor 210, a memory 220, a wireless communication module 230, a display screen 240, and a power supply 250.

[0140] It should be understood that the structure illustrated in this application does not constitute a specific limitation on the second device 200. In other embodiments of this application, the second device 200 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0141] In this embodiment, the hardware and software included in the processor 210 and the functions of each hardware and software are similar to those of the processor 110. This application will not be described in detail here. The processor 210 can be used to call the display screen 240 to display the projection interface and security interface from the first device 100. In some embodiments, the processor 210 can also be used to generate an operation instruction, which is used to instruct the user to operate the APP on the first device 100.

[0142] The communication solution provided by the wireless communication module 230 is similar to that provided by the wireless communication module 160 and will not be described in detail here. The communication solution provided by the wireless communication module 230 enables the second device 200 to communicate with the first device 100, thereby enabling interactive functions of video streams and instructions between the second device 200 and the first device 100. For example, the wireless communication module 230 enables the second device 200 to receive the video stream to be projected from the first device 100.

[0143] The memory 220 can be used to store one or more computer programs, each of which includes instructions. The processor 210 can execute the instructions stored in the memory 220, thereby causing the second device 200 to perform the display method provided in some embodiments of the present application. The memory 220 can also be used to cache data involved in the second device 200 performing the display method, such as data of the security interface.

[0144] The display screen 240 is used to display the controls, information, images, videos, etc. of the second device 200, as well as the projection interface information from the first device 100. In other embodiments, when the second device 200 receives a security protection instruction from the first device 100, the display screen 240 can be triggered to display a security interface at the forefront of the projection interface corresponding to the security protection instruction. In this embodiment, the hardware composition of the display screen 240 is similar to that of the aforementioned display screen 194 and is not described in detail here.

[0145] The power supply 240 can be used to provide power to the processor 210 , the memory 220 , the display screen 240 , etc.

[0146] The UI for realizing the display function involved in the embodiment of the present application is a medium interface for interaction and information exchange between an application or an operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is a source code written in a specific computer language such as Java and Extensible Markup Language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as video, picture, text, button and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as <head> and <body>. A node corresponds to a control or attribute in the interface, and the node is presented as user-visible content after parsing and rendering. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is a source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code, such as HTML through <body>. 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0147] A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operations that uses graphics. It can be an icon, window, control, or other interface element displayed on the display of an electronic device. Controls can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.

[0148] It should be pointed out that in a typical security protection mechanism based on the screen projection scenario, the first device 100 projects the screen to the second device 200, usually in units of display, and the second device 200 displays the security interface in units of window interface. Based on this, if the display projected by the first device 100 contains at least two window interfaces, the second device 200 can display at least two corresponding projection interfaces. When any of the at least two window interfaces requires security protection, the second device 200 displays the security interface in units of window interface, so that the at least two projection interfaces display the security interface or a black screen (at this time, the display data of the first device will not be transmitted to the second device). In this way, users cannot operate the projection interface normally even if it has no security requirements, and the user experience is poor.

[0149] In order to solve the above problems, the present invention provides a display method and an electronic device. Figure 4A As shown, the display method involved in the embodiment of the present application includes: the first device receives a screen projection instruction input by the user. In response to the screen projection instruction, the first device projects the first window interface to the second device, and accordingly, the second device displays the screen projection interface of the first window interface. Furthermore, after the first device responds to the instruction to run the second window interface, it projects the second window interface to the second device (this instruction comes from the second device, for example, when the first window interface is the main interface, the instruction is the operation of double-clicking to run the application). After traversing the Window state, the first device confirms that the second window interface contains a security flag, and sends the task number and security protection instruction of the second window. The second device displays the screen projection interface and security interface of the first window interface (the security interface is used to replace the original second window interface display).

[0150] Among them, the trigger for the first device to traverse the Window state may be a periodic trigger, or it may be triggered by a change in the position of the Activity in the stack, or it may be triggered by a change in the Activity at the top of the stack.

[0151] Based on this, Figure 4A In the illustrated implementation scenario, after the first device starts the second window interface, the flag corresponding to the Activity of the second window interface presented in response to the startup instruction is 1. For example, the second window interface is the login interface of a bank APP. After the second window is started, the security keyboard is triggered. In this way, before the first device projects the screen, it immediately determines that the second window contains a security flag. Then, the first device sends the task number and security protection instruction of the second window to the second device, so that the second device displays the projection interface and security interface of the first window interface.

[0152] In some other embodiments, the present application also provides another exemplary display method. Figure 4A As shown, in this embodiment, the first device receives a screen projection instruction input by the user. In response to the screen projection instruction, the first device projects the first window interface to the second device. After the first device responds to the instruction to run the second window interface, it determines within the first traversal cycle that the second window interface does not contain a security flag, and then projects the second window interface to the second device. Furthermore, in the subsequent traversal cycle, the first device detects that the second window interface contains a security flag, and then the first device sends the task number and security protection instruction of the second window interface to the second device. Correspondingly. The second device displays the screen projection interface and security interface of the first window interface.

[0153] Figure 4B In the illustrated implementation scenario, after the first device starts the second window interface, the content of the currently started second window interface has no security requirements, and accordingly, the current Window state flag of the second window interface is set to 0. Then, the first device can project the content of the second window interface to the second device, so that the second device displays the projection interface of the first window interface and the projection interface of the second window interface. In response to the user's trigger, the second window interface changes, thereby triggering the Activity change of the changed interface, so that the changed second window interface has security requirements, and the corresponding Windowstate flag is set to 1. Furthermore, the first device can determine that the second window interface at this time contains a security flag, and the first device triggers the second device to display the second window interface as a security interface, so that the second device displays the projection interface and security interface of the first window interface.

[0154] For example, after the second window interface is started, the main interface of the wallet APP is displayed. The main interface of the wallet APP is, for example, a photo without security requirements, so the first device projects the wallet APP to the second device. The second device can display the projection interface of the wallet APP main interface. Furthermore, in response to the user's trigger, the second window interface enters the payment interface of the wallet APP. At this time, the second window interface includes a payment password input control, which is an interface with security requirements. After detecting the security flag, the first device sends the second window interface (the interface of the wallet APP) task number and security protection instructions to the second device, so that the second device displays the projection interface of the second window interface as a secure interface.

[0155] Furthermore, the above Figure 4A or Figure 4B In the illustrated embodiment, if the second window interface further updates its displayed content in response to a user's operation instruction, and the corresponding content has no security requirements, the corresponding window state flag is set to 0. Then, during the periodic traversal of the window state, the first device can determine based on this window state that the second window interface has no security flag. Furthermore, the first device can project the updated second window interface to the second device. Correspondingly, the second device hides the security interface and displays the updated projection interface of the second window interface.

[0156] According to the above description of screen projection, the content of the screen projection from the first device to the second device is the window interface contained in the display of the first device, that is, the first window interface and the second window interface are the window desktop contained in the display of the first device. It should be understood that if the display of the first device also contains a third window interface and a fourth window interface, etc., the first device should also detect whether the third window interface and the fourth window interface contain a security flag. After that, the first device sends the projection information of the third window interface and the fourth window interface to the second device according to the detection results. It is not repeated here. In addition, in an embodiment of the present application, the first device performs a screen projection each time the display of the first device is updated. Among them, the increase, decrease, and change of any window interface in the display are all considered to be display updates.

[0157] It should be noted that the first device projects the screen to the second device, which can be achieved by establishing a network connection between the first device and the second device, and then the first device sends the interface to be displayed to the second device in the form of a video stream. Afterwards, the second device can cut the video stream based on the parameter information sent by the first device (such as the size of the window interface, the position of the window interface, and the relative position between the window interfaces), and then display each interface of the screen projected by the first device in the form of a window. Based on this, in the implementation scenario of screen projection, the WMS is configured to tile the display of the window interface to be projected, and the AMS adds a taskID to each window interface to be projected, so that the first device generates a video stream corresponding to each window interface to be projected, and identifies each window interface to be projected.

[0158] It should be noted that the display includes the window interfaces of all APPs running on the first device, and the window interfaces to be projected can be all the window interfaces in the display or part of the window interfaces in the display.

[0159] In some embodiments, the first device determines whether each window interface has security requirements before screen projection. Then, based on the detection results, the first device sends the taskID of each window interface and the projection information of the window interface to the second device. The second device displays the security interface and the projection interface containing the window interface content based on the taskID, wherein the security interface is the projection interface of the window interface with security requirements. In this way, the second device can not only safely display the interface with security requirements, but also ensure that the user can operate normally on the interface without security requirements.

[0160] For ease of understanding, the following examples of this application will be described with Figure 2A and Figure 2B As an example, take the mobile phone with Figure 2C Taking the laptop computer with the shown structure as an example, the display method in the screen projection scenario provided by the embodiment of the present application is specifically explained in combination with the accompanying drawings.

[0161] The embodiment of the present application can divide the actual implementation process into the following three stages: the stage of starting the screen projection function, the display stage in the security requirement scenario, and the display stage after the security interface operation. The following introduces the embodiment of the display method in combination with these three stages.

[0162] Stages for starting the screen projection function:

[0163] In an embodiment of the present application, a mobile phone and a laptop computer can establish a communication connection through their respective wireless communication modules. After receiving the instruction input by the user to turn on the screen projection function, the mobile phone can, for example, send the video stream that needs to be projected to the laptop computer based on wireless direct connection (WiFiDirect) technology to realize the screen projection function. In some embodiments, the mobile phone and the laptop computer can establish a communication connection through their respective NFC modules or Bluetooth modules. After receiving the instruction from the user to click to turn on "multi-device collaboration", the screen projection function of the mobile phone is turned on. In a scenario where the distance between the mobile phone and the laptop computer is less than or equal to the communication transmission distance, the mobile phone projects the first group of window interfaces to the laptop computer. If the scenario requirements of the aforementioned screen projection are met, it is considered that the mobile phone has received the screen projection instruction. It will not be described in detail here. The display interface related to the first group of window interfaces projected by the mobile phone to the laptop computer is as follows: Figures 5A to 5C-2 shown.

[0164] It is understandable that the aforementioned modules for establishing a communication connection between the mobile phone and the laptop computer, as well as the interface for enabling the mobile phone screen projection function, are schematic descriptions and do not constitute a limitation on the embodiments of the present application. In other embodiments, the mobile phone and the laptop computer can also establish a connection based on other communication modules. In addition, the interface for enabling the mobile phone screen projection function may vary depending on the model of the mobile phone and the version of the mobile phone operating system, which is not limited here.

[0165] like Figure 5A As shown, Figure 5A GUI 51 is an exemplary first display interface of the laptop after the mobile phone screen is projected onto the laptop. GUI 51 includes projection interface 510, which is the projection interface of the mobile phone desktop and includes images of various interface elements on the mobile phone desktop, such as the gallery, mailbox, wallet, etc.

[0166] Figure 5A In the corresponding embodiment, when the screen projection instruction is received, regardless of what interface is displayed on the mobile phone display screen and what app is running in the foreground of the mobile phone, the mobile phone calls the wireless communication module to send the video stream of the mobile phone desktop to the laptop computer. The laptop computer then decodes the corresponding video stream and displays the screen projection interface 510 corresponding to the video stream. At this time, the apps running on the mobile phone can optionally run in the form of free floating windows.

[0167] In some embodiments, the aspect ratio of the projection interface can be the same as the aspect ratio of the window interface. The size of the projection interface depends on the size of the second device screen. For example, if the second device screen size is 285.7 mm wide by 214.3 mm high, the size of the projection interface can be 110 mm wide by 210 mm high. If the second device screen size is 1456.4 mm wide by 850.9 mm high, the size of the projection interface can be 450 mm wide by 800 mm high.

[0168] like Figure 5B As shown, Figure 5B The illustrated GUI52 is an exemplary display interface of the mobile phone during the screen projection process. Figure 5B GUI53 is a second exemplary display interface of the laptop after the mobile phone screen is projected onto the laptop. GUI52 is, for example, the display interface of the Gallery app running on the mobile phone, and GUI52 includes image 521 and image 522. GUI53 includes projection interface 530, which is the projection interface of GUI52. Projection interface 530 includes projection image 531 of image 521 and projection image 532 of image 532.

[0169] Figure 5B In the corresponding embodiment, during the period when the screen projection instruction is received, the mobile phone is displaying a window interface. Then, the WMS traverses whether the Activity corresponding to the window interface contains a flag. If the Activity corresponding to the window interface does not contain a flag, the GPU of the mobile phone can render the video stream of the interface currently being displayed by the mobile phone. Afterwards, the mobile phone calls the wireless communication module to send the video stream of the interface being displayed by the mobile phone to the laptop. Then, the laptop decodes the corresponding video stream and displays the screen projection interface 530 corresponding to the video stream.

[0170] In other embodiments, if the activity corresponding to the window interface includes a flag, the mobile phone can project the mobile phone's desktop to the laptop. Accordingly, the laptop's GUI can be shown as GUI51. After the mobile phone receives the user's operation and updates the window interface, the mobile phone can project the updated window interface to the laptop. See the description of the relevant embodiments below for details, which will not be detailed here.

[0171] In other embodiments, if the mobile phone does not display any APP interface, the mobile phone will project the mobile phone desktop to the laptop computer. Accordingly, the GUI of the laptop computer may be as shown in GUI51.

[0172] like Figure 5C As shown, Figure 5C The illustrated GUI54 is another exemplary display interface of the mobile phone during the screen projection process. Figure 5C The illustrated GUI55 is a third exemplary display interface of the laptop after the mobile phone is projected onto the laptop. Among them, GUI54 is, for example, a display interface of a window interface of a mobile phone that is running and has security requirements. An exemplary display interface of GUI54 is described in detail in Figure 5C-2 GUI55 includes a screen projection interface 551 and a screen projection interface 552. Among them, the screen projection interface 551 is the screen projection interface of the window interface of the mobile phone that is running and has no security requirements, and the screen projection interface 552 is the screen projection interface of the mobile phone desktop.

[0173] Figure 5C In the corresponding embodiment, during the period of receiving the screen projection instruction, the mobile phone is running several APPs. In this embodiment, the mobile phone can respectively detect whether the window interfaces of the running APPs contain security flags. After that, the mobile phone can project the window interfaces that do not contain the security flags and the mobile phone's desktop to the laptop, and the window interfaces that contain the security flags (such as the security flags) are stacked and displayed on the mobile phone's display screen. Figure 5C GUI54 in the figure) so that the user can perform the next operation. The laptop displays a projection interface of a window interface without security requirements (such as Figure 5C 55 in FIG.

[0174] Exemplarily, during the period of receiving the screen projection instruction, the mobile phone is running three APPs, namely, mailbox, wallet and gallery, wherein the current window interface of the mailbox is, for example, the login interface, and the current window interface of the wallet is, for example, the interface for managing bank cards. Then, WMS traverses the Activities corresponding to the three window interfaces to determine whether each Activity contains a flag. For example, WMS determines that the Activity corresponding to the current window interface of the mailbox contains the first flag, the Activity corresponding to the current window interface of the wallet contains the second flag, and the Activity corresponding to the current window interface of the gallery does not contain a flag. Afterwards, WMS can report the results of the traversal to DMS. DMS configures the display of the mobile phone and the display strategy of each window interface based on the traversal results. The display interface of the display is as follows: Figure 5C-1 shown.

[0175] like Figure 5C-1 As shown, Figure 5C-1 The illustrated display50 includes display501 and display502. Among them, display501 is deployed on the display screen of the mobile phone and is a display visible to the user. Display502 is an extended display of display501 and is a virtual display invisible to the user. In this embodiment, there are no security requirements for the mobile phone's desktop and the window interface of the gallery, so the mobile phone can project the desktop and the window interface of the gallery to the laptop. Based on this, the AMS can configure taskIDs for the mobile phone's desktop and the window interface of the gallery respectively. For example, the AMS configures task 0 for the mobile phone's desktop and the AMS configures task 1 for the window interface of the gallery. In addition, the DMS can configure the mobile phone's desktop to be presented on display501, and configure the window interface of the gallery to be presented on display502. Furthermore, the mobile phone's GPU can render the video streams of the mobile phone's desktop and the window interface of the gallery. Afterwards, the mobile phone calls the wireless communication module to send the video streams of task 0 and the mobile phone's desktop, as well as the video streams of task 1 and the window interface of the gallery to the laptop. Then, the laptop decodes the corresponding video stream, segments the decoded image, and displays it. Figure 5C GUI55 in GUI55. The projection interface 551 in GUI55 is, for example, the projection interface of the mobile phone desktop, and the projection interface 551 corresponds to task 0. The projection interface 552 is, for example, the projection interface of the window interface of the gallery, and the projection interface 552 corresponds to task 1.

[0176] Furthermore, the window interface of the mailbox and the window interface of the wallet are interfaces with security requirements. Even if they are projected to a laptop, the laptop is not allowed to display the contents of the two window interfaces. Based on this, the mobile phone can display the two window interfaces on the mobile phone's display screen instead of projecting the two window interfaces to the laptop. Among them, the DMS can configure the display strategy of the two window interfaces on the mobile phone's display screen based on the traversal results of the WMS. After that, the DMS can display the two window interfaces on the mobile phone's display screen through the display driver and surface manager. The display interface of the two window interfaces is as follows: Figure 5C-2 shown.

[0177] Figure 5C-2 GUI 56 is an exemplary display interface of GUI 54. GUI 56 displays an image of window interface 561 and an image of window interface 562, which are displayed stacked. GUI 56 also includes a delete icon 563 at the bottom. Window interface 561, for example, is the login interface for a mailbox, while window interface 562 is the bank card management interface for a wallet. Upon receiving a user instruction to click delete icon 563, the AMS terminates both the mailbox and wallet processes. Upon receiving a user instruction to click the image of window interface 561, the DMS controls the mailbox login interface to be displayed at the forefront of the phone's display screen, allowing the user to enter their mailbox account and password. Similarly, upon receiving a user instruction to click the image of window interface 562, the DMS controls the bank card management interface for the wallet to be displayed at the forefront of the phone's display screen, allowing the user to perform the next action, such as a payment. This will not be further described here.

[0178] In some embodiments, the mobile phone can Figure 5C-1 After the display 50 is projected onto the laptop computer, the GUI 56 is displayed.

[0179] Furthermore, after the phone projects display 50 onto the laptop, the phone displays GUI 56 and other subsequent displays, triggering an update of the window interface. Accordingly, display 50 is updated to another display. The WMS then checks whether the window interface in the updated display contains a security flag. Based on the detection result, the phone can project the updated display.

[0180] With this implementation, after receiving a screen casting command, the phone selects a group of screens without security requirements as the first group of screens to be cast to the laptop, regardless of whether the window interface of the currently running app has security requirements. This allows at least one operational interface to be cast to the laptop, allowing the user to continue using it, while also ensuring the security of the windows with security requirements and saving transmission resources.

[0181] It should be noted that in some possible implementations, the mobile phone can send the security interface to the laptop at this stage. Accordingly, the laptop can store the security interface locally and, after receiving the security protection instruction from the mobile phone, instruct the security protection instruction to display the screen projection interface as the security interface.

[0182] Figure 6 A security interface 60 is illustrated. The security interface 60 includes an icon 61 and an icon 62. Icon 61, for example, displays the following message: "Operation is required on the mobile phone to protect your privacy." Icon 62, for example, displays the following message: "Display this application on the mobile phone." Icon 62 optionally receives clicks from the user on the computer and sends them to the mobile phone side through, for example, the Miracst UIBC (user inputback control) protocol. The background color of the security interface 60 can be a solid color, such as black. In other embodiments, in order to optimize the user's visual experience, the background of the security interface 60 can be a Gaussian blurred mobile phone wallpaper.

[0183] In some embodiments, the background of the security interface can be optionally generated by the computer. In some embodiments, in order to better provide users with a consistent experience on the phone and computer, the security interface can be optionally generated by the phone. For example, the phone performs Gaussian blur processing on its wallpaper and sends it to the computer, which stores the processed wallpaper. When the computer receives the security interface sent by the phone, it reads the wallpaper locally and displays it. It is understandable that when the phone detects that the wallpaper has been changed, the phone will perform the above operations to ensure that the background of the computer and security interface can be consistent with the wallpaper on the phone.

[0184] It is understandable that Figure 6 This is merely an illustrative description and does not limit the embodiments of the present application. In other embodiments, the security interface may further include other information. In addition, the information included in the security interface may be flexibly configured. This will not be described in detail here.

[0185] The security requirements interface appears:

[0186] After the phone projects the first set of windows onto the laptop, any continued user activity on either the phone or the laptop could trigger the display of a security-required interface. The following describes an embodiment of displaying a security interface from two perspectives: a user triggering the screen on the phone and a user triggering the screen on the laptop.

[0187] Scenario 1: User operating on a laptop

[0188] In an embodiment of the present application, after the laptop computer receives an instruction from the user to click on the screen projection interface, the laptop computer can generate a first operation instruction, and the first operation instruction is associated with the first APP. The first APP is the APP indicated by the image clicked by the user. Afterwards, the laptop computer sends the first operation instruction to the mobile phone based on WiFi Direct technology. The mobile phone responds to the first operation instruction and performs an operation on the first APP, thereby triggering a change in the window interface displayed by the first APP, and then, updating the display of the mobile phone. Then, the mobile phone performs a security flag detection and re-projection operation on the corresponding updated display, so that the laptop computer displays the screen projection interface of the APP triggered by the user, realizing the function of the user operating the APP of the mobile phone based on the screen projection interface. Figure 7 (a) to (b) in FIG. 1 illustrate an exemplary display method in scenario 1. Figure 8 (a) to (b) show the Figure 7 Schematic implementation process, schematic change process of mobile phone display.

[0189] like Figure 7 As shown in (a), GUI70 is a mobile phone Figure 8 The display interface of the laptop computer after the display 80 shown in (a) is projected onto the laptop computer. GUI70 includes a projection interface 701 and a projection interface 702. The projection interface 701 is, for example, the projection interface of the mobile phone desktop, including images of APPs such as mailboxes and memos. The projection interface 702 is, for example, the projection interface of the window interface of the memo. Among them, the projection interface 701 corresponds to the identifier task 0, and the projection interface 702 corresponds to the identifier task 1. After receiving the user's instruction to click on the mailbox, the laptop computer generates a first operation instruction, which is used to trigger the mobile phone to run the mailbox APP (an instance of the aforementioned first APP). Afterwards, the laptop computer sends the first operation instruction to the mobile phone. The mobile phone responds to the first operation instruction to run the mailbox APP, thereby starting the login interface of the mailbox. The login interface of the mailbox is as shown Figure 12 As shown, it is not described in detail here. Then, the display80 on the mobile phone is updated to Figure 8 Afterwards, the mobile phone projects the display 81 to the laptop, so that the GUI 70 of the laptop is updated to Figure 7 GUI71 is shown in (b).

[0190] After receiving the user's click operation, the laptop computer can identify the first app identifier corresponding to the image clicked by the user. The first app identifier is used to indicate the first app. The laptop computer then generates a first operation instruction, which includes the first app identifier. Furthermore, the mobile phone can determine the first app based on the first app identifier. The first app identifier can be, for example, the name of the first app.

[0191] Figure 7 GUI71 shown in (b) is a mobile phone Figure 8 The display interface of the laptop computer after the display 81 shown in (b) is projected onto the laptop computer. GUI71 includes a projection interface 711, a projection interface 712, and a projection interface 713. The projection interface 711 corresponds to the identifier task 0, the projection interface 712 corresponds to the identifier task 1, and the projection interface 713 corresponds to the identifier task 2. Among them, the projection interface 711 is, for example, the projection interface of the mobile phone desktop, and the projection interface 711 is the same as the projection interface 701 in GUI70. The projection interface 712 is, for example, the projection interface of the window interface of the memo, and the projection interface 712 is the same as the projection interface 702 in GUI70. The projection interface 713 is, for example, the projection interface of the window interface of the mailbox, and the projection interface 713 is as follows. Figure 6 The security interface shown is not described in detail here.

[0192] like Figure 8 As shown in (a), display80 includes window interface 71 and window interface 72. Window interface 71 is the mobile phone desktop, and the screen is cast to obtain the screen cast interface 701 in GUI70. Window interface 72 is the window interface of the memo, and the screen is cast to obtain the screen cast interface 702 in GUI70. Due to the display of window interface 71 and window interface 72, WMS expands a virtual display based on the width of the mobile phone display screen, so that display80 includes display801 and display802, display801 is the display area corresponding to the display screen of the mobile phone, display802 is a virtual display, and the width of display802 can be the same as the width of display801. Furthermore, WMS deploys window interface 71 in display801 for display, and deploys window interface 72 in display802 for display. In addition, in order to facilitate the identification of window interface 71 and window interface 72, AMS sets the identifier task 0 for window interface 71 and the identifier task 1 for window interface 72. Furthermore, in this embodiment, after determining that window interface 71 and window interface 72 have no security requirements, the mobile phone sends task 0, task 1 and the video streams of window interface 71 and window interface 72 to the laptop, causing the laptop to present GUI 70.

[0193] Furthermore, after the mobile phone responds to the first operation instruction to open the login interface of the mailbox, a window interface is added to the display of the mobile phone based on display80, and then, display80 is updated to display81.

[0194] like Figure 8 As shown in (b), display81 is a schematic display interface of GUI71 obtained by projection. Display81 includes window interface 71, window interface 72 and window interface 73. Window interface 71 is projected to obtain projection interface 711 in GUI71. Window interface 72 is projected to obtain projection interface 712 in GUI71. Window interface 73 is projected to obtain projection interface 713 in GUI71. In this embodiment, in order to accommodate window interface 71, window interface 72 and window interface 73, DMS expands a virtual display based on the display screen of the mobile phone to obtain display81. Display81 includes display811 and display812. Display811 is the display screen of the mobile phone, display812 is a virtual display, and the width of display812 can be twice the width of display811. Furthermore, WMS deploys window interface 71 in display 811 for display, and sets the identifier task 0 for window interface 71, and deploys window interface 72 and window interface 73 in display 812 for tiled display, and sets the identifier task 1 for window interface 72 and the identifier task 2 for window interface 73.

[0195] In some embodiments, display 81 may be obtained by the mobile phone continuing to operate display 80 .

[0196] Then, the mobile phone's WMS traverses the activity of window interface 71, the activity of window interface 72, and the activity of window interface 73 to determine whether each activity contains a flag (which can be understood as determining whether the flag is set). Exemplarily, the WMS determines that neither the activity of window interface 71 nor the activity of window interface 72 contains a flag, while the activity of window interface 73 contains the first flag. Afterwards, the mobile phone sends task 0, task 1, task 2, the video streams of window interface 71 and window interface 72, and security protection instructions to the laptop. Among them, the video stream of window interface 71 corresponds to task 0, the video stream of window interface 72 corresponds to task 1, and the security protection instruction corresponds to task 2. Then, the laptop splits the video streams of window interface 71 and window interface 72, and displays the projection interface 711 and projection interface 712. In addition, the laptop responds to the security protection instruction and displays the projection interface of window interface 73 as a security interface to obtain projection interface 713.

[0197] In some embodiments, after the mobile phone projects the first set of window interfaces onto the laptop, the mobile phone's display screen may always display the mobile phone desktop. Window interfaces displayed by the mobile phone in response to operating instructions from the laptop may be displayed on the virtual display.

[0198] Scenario 2: User operation on mobile phone

[0199] In the embodiment of the present application, after the mobile phone receives the second operation instruction input by the user, the second APP interface triggered by the instruction is usually displayed on the mobile phone desktop. In this way, the mobile phone desktop is updated, and the mobile phone display is also updated accordingly. Then, the mobile phone can project the screen to the laptop based on the updated display, so that the laptop screen projection interface is updated accordingly. Figure 9 (a) to (c) in FIG. 1 illustrate an exemplary display method in scenario 2. Figure 10 (a) to (b) show the Figure 9 Schematic implementation process, schematic change process of mobile phone display.

[0200] like Figure 9 As shown in (a), GUI90 is a mobile phone that Figure 10 GUI 90 includes a projection screen interface 901 and a projection screen interface 902. Projection screen interface 901 is, for example, a projection screen interface of a mobile phone desktop. Projection screen interface 902 is, for example, a projection screen interface of a memo window interface. Projection screen interface 701 corresponds to task 0, and projection screen interface 702 corresponds to task 1.

[0201] like Figure 9 As shown in (b), GUI91 is the desktop of the mobile phone, and GUI91 includes interface elements such as mailboxes. After display101 is projected onto the laptop, the mobile phone receives a second operation instruction input by the user, and the second operation instruction is used to start a second APP. The second APP is, for example, a mailbox APP. Furthermore, since the user needs to operate the mailbox on the mobile phone, in this embodiment, the mobile phone displays the login interface of the mailbox on the display screen. The login interface of the mailbox is as follows: Figure 12 As shown, it will not be described in detail here. Correspondingly, the window interface corresponding to the mobile phone desktop is updated to the mailbox login interface, and display101 is updated to Figure 10 Afterwards, the mobile phone projects the display 102 to the laptop, so that the GUI 91 of the laptop is updated to Figure 9 GUI92 is shown in (c).

[0202] Figure 9 The GUI 92 shown in (c) is a mobile phone that Figure 10 After the display 102 shown in (b) is projected onto the laptop, the laptop's display interface is displayed. GUI 92 includes a projection interface 921 and a projection interface 922. The projection interface 921 is a projection interface of the mailbox window interface, for example, Figure 6 The security interface shown. The projection interface 922 is, for example, a projection interface of the memo window interface. Among them, the projection interface 921 corresponds to the identifier task 0, and the projection interface 922 corresponds to the identifier task 1.

[0203] like Figure 10 As shown in (a) of FIG, display 101 includes window interface 1011 and window interface 1012. DMS sets the identifier task 0 for window interface 1011 and the identifier task 1 for window interface 1012. Window interface 1011 is projected to obtain projection interface 901 in GUI 90. Window interface 1012 is projected to obtain projection interface 90 in GUI 90. Figure 8 It is similar to display80 shown in (a) and will not be described in detail here.

[0204] like Figure 10 As shown in (b), display102 includes window interface 1021, window interface 1022 and window interface 1023. Among them, window interface 1021 is the same as window desktop 1011, which is the desktop of the mobile phone, window interface 1022 is the same as window interface 1012, which is the interface of the memo, and window interface 1023 is the login interface of the mailbox, corresponding to the identifier task 2. That is, after the mobile phone responds to the second operation instruction to display the login interface of the mailbox, the AMS starts the mailbox APP, enters the mailbox login interface, and configures the label task2 for the mailbox APP. DMS performs expansion on the basis of display101 to obtain display102 so as to accommodate window interface 1023. Furthermore, in this embodiment, the mobile phone can cast window interface 1022 and window interface 1023 to a laptop computer. In specific implementation, the mobile phone can determine through WMS that window interface 1023 is an interface with security requirements, while window interface 1022 has no security requirements. Afterwards, the mobile phone sends task 1, task 2, the security protection instruction, and the video stream of the window interface 1012 to the laptop computer. Task 2 corresponds to the security protection instruction, and task 1 corresponds to the video stream of the window interface 1012. Then, the laptop computer displays GUI 92.

[0205] Based on Figure 9 and Figure 10 The same principle of the embodiment shown in FIG. Figure 5C-2 In the corresponding embodiment, when the mobile phone receives a user instruction to trigger the display of window interface 561, or when the user triggers the display of window interface 561, the mobile phone's desktop interface is updated to a window interface that meets security requirements. Then, the mobile phone projects the screen again based on the updated window interface, so that the screen projection interface corresponding to task 0 on the laptop is a secure interface.

[0206] In other embodiments, if the interface displayed by the mobile phone in response to the second operation instruction does not require security, the mobile phone's display screen displays a window interface that does not require security. Furthermore, the mobile phone can project the image of the corresponding window interface to the laptop computer. Accordingly, the laptop computer can display the projection interface of the corresponding window interface and the projection interface of the memo window interface.

[0207] It can be seen that by adopting this implementation method, the mobile phone associates the window interface to be projected with the task ID. After the window interface is updated, the mobile phone can instruct the laptop to display the projection interface of each window interface through the task ID, so that the laptop can display the projection interface in units of window interfaces, which can not only safely display the interface with security requirements, but also ensure that users can operate normally on the interface without security requirements.

[0208] Display phase after operating the security interface:

[0209] In the first scenario described above, after the laptop displays the first app's projection interface as a secure interface, the user can use the laptop to trigger the corresponding window interface to be displayed on the phone's display, allowing the user to continue operating the first app on the phone. The display of the first app's window interface on the phone's display, as well as continued user operation, will trigger an update to the phone's display, which in turn triggers a change in the projection interface displayed on the laptop.

[0210] The following combination Figure 11 and Figure 12 , describes the display method of the laptop computer and the change process of the mobile phone display in the scenario where the window interface of the first APP is displayed on the mobile phone display.

[0211] Continuing from the above scenario, Figure 11 The GUI of the laptop in (a) is, for example, GUI71, wherein the screen projection interface 713 includes an icon 1101, and the icon 1101 is connected to the screen projection interface 713. Figure 6 After receiving the user's click on icon 1101, the notebook computer generates a third operation instruction, which is used to trigger the mobile phone to display the Figure 12 The login interface of the mailbox is shown in the figure, so that the user can log in to the mailbox on the mobile phone. Then, the window interface 73 in the display 81 of the mobile phone is displayed on the display screen of the mobile phone, so that the mobile phone appears Figure 12 Then, the mobile phone can project the window interface 72 and the window interface 73 to the laptop again based on the display 81, so that the GUI 71 of the laptop is updated to Figure 11 GUI111 is shown in (b).

[0212] After receiving the user's click operation, the laptop can obtain the identifier task 2 corresponding to the screen projection interface 713. Thereafter, the laptop generates a third operation instruction, which includes the identifier task 2. Furthermore, the mobile phone can determine the mailbox login interface based on the identifier task 2.

[0213] Figure 11 The GUI 111 shown in (b) is a mobile phone that Figure 8 After the display 81 shown in (b) is projected onto the laptop, the laptop's display interface is displayed. GUI 111 includes a projection screen interface 1111 and a projection screen interface 1112. The projection screen interface 1111 corresponds to the identifier task 0, and the projection screen interface 1112 corresponds to the identifier task 1. Among them, the projection screen interface 1111 is the projection screen interface of the mailbox login interface, and the projection screen interface 1111 is as follows. Figure 6 The projection interface 1112 is the projection interface of the memo window interface, and the projection interface 1112 is the same as the projection interface 712 in the GUI 71 and will not be described in detail here.

[0214] Combine Figure 8 As shown in (b) of the figure, in this embodiment, the mobile phone can obtain and render the video stream of the window interface 72 based on the starting coordinates of the upper left corner of the window interface 72, and then send task 1, task 2, the rendered video stream and the security protection instruction to the laptop computer, wherein task 1 corresponds to the video stream and task 2 corresponds to the security protection instruction. Then, the laptop computer displays the projection interface 1112 corresponding to task1 and the projection interface 1111 corresponding to task2. It should be pointed out that in this embodiment, the projection information sent by the mobile phone does not include task0 in display81 and the information corresponding to task0 (i.e., the interface of the mobile phone desktop), so the projection interface corresponding to task 0 is not included in GUI111. From the user's visual effect, during the process of updating GUI71 to GUI111, the laptop computer deleted the projection interface 713 and updated the projection interface 711 to the security interface.

[0215] Figure 12 GUI 120, shown schematically, is a mobile phone's email login interface. GUI 120 includes a control 1201 for entering an "email account," a control 1202 for entering a "password," and a "Login" button 1203. Control 1202 includes a reminder message "Enter Password" in a lower grayscale font to remind the user to enter their email login password. After receiving the password entered by the user, the "Enter Password" prompt is hidden. In some embodiments, control 1201 includes a reminder message "Enter Email Account" in a lower grayscale font to remind the user to enter their email login account. After receiving the user's input, the "Enter Email Account" prompt is hidden. In other embodiments, control 1201 includes the email login account, and control 1202 includes the reminder message "Enter Password." When the mobile phone displays GUI 120 on its display, the phone's display is updated from display 81 to display 142.

[0216] Furthermore, the following combination Figures 13 to 14B , describes the display method of the laptop computer and the change process of the mobile phone display after the mobile phone receives the user's operation on the first APP.

[0217] It should be pointed out that Figure 11 The implementation scenario corresponding to (b) in Figure 9 Based on this, the following embodiments of the display method are applicable to both of the above two implementation scenarios, and the embodiments of this application will not be repeated.

[0218] Combine Figure 12 , the mobile phone can receive the login instruction input by the user by clicking button 1203, and the login instruction includes the email account and password input by the user. After confirming that the corresponding email account and password are correct, AMS calls the Activity of the mailbox main interface, and accordingly, the mobile phone displays the mailbox main interface. The mailbox main interface is as follows Figure 13 As shown in the window interface marked with task 2.

[0219] Figure 13 The illustrated display 142 includes window interface 1421, window interface 1422 and window interface 1423. Window interface 1421 corresponds to task 0, which is the mobile phone desktop. Window interface 1422 corresponds to task 1, which is the window interface of the memo. Window interface 1423 corresponds to task 2, which is the window interface of the mailbox main interface. In this embodiment, after confirming that the mailbox account and password are correct, DMS can control the login interface corresponding to task 0 in display 81 to be closed or hidden, and display the mailbox main interface through a new window interface. In this way, the window interface identified by task 0 is updated to the mobile phone desktop, and a new window interface is added. Further, AMS configures the new window interface with task 2 to obtain Figure 13 Schematic display142.

[0220] In some embodiments, the mobile phone can project the window interface 1422 and the window interface 1423 in the display 142 to the laptop computer to obtain Figure 14A In some other embodiments, the mobile phone can project all the window interfaces of display 142 to the laptop computer to obtain Figure 14B Schematic GUI 152.

[0221] like Figure 14A As shown, GUI 151 includes a projection interface 1511 and a projection interface 1512. Among them, projection interface 1511 corresponds to task 1 and is the projection interface of window interface 1422 in display 142. Projection interface 1512 corresponds to task 2 and is the projection interface of window interface 1423 in display 142.

[0222] like Figure 14B As shown, GUI 152 includes projection interface 1521, projection interface 1522, and projection interface 1523. Projection interface 1521 corresponds to task 0 and is the projection interface of window interface 1421 in display 142. Projection interface 1522 corresponds to task 1 and is the projection interface of window interface 1422 in display 142. Projection interface 1523 corresponds to task 2 and is the projection interface of window interface 1423 in display 142.

[0223] It should be pointed out that the interaction of various information and commands between the above-mentioned mobile phone and laptop computer can be based on WiFi Direct technology.

[0224] It is understandable that Figures 5A to 14B This is only a schematic description and does not limit the embodiments of the present application. In other embodiments, the embodiments of the above display method are also applicable to the display of other APPs on mobile phones, and will not be described one by one here. In addition, the GUIs in the above scenarios may vary depending on the model of the mobile phone, the version of the system installed, etc., and will not be described in detail here.

[0225] In addition, the above embodiments are only described using mobile phones and laptop computers as examples and do not limit the embodiments of the present application. In other embodiments, the first device involved in the embodiments of the present application can also be implemented as a foldable screen device, a tablet computer, etc., and the second device involved in the embodiments of the present application can also be implemented as a smart screen, etc. This is not limited here.

[0226] The following describes the overall process of the display method involved in the embodiment of the present application by taking the first device implemented as a mobile phone and the second device implemented as a PC as an example.

[0227] See also Figure 15 , Figure 15 The third exemplary display method of the present application is illustrated. In some embodiments, multi-screen collaboration is performed on the mobile phone side. The mobile phone side first determines whether it is currently multi-screen collaboration. If not, the default application lock policy is adopted (that is, the mobile phone side will not send the video stream to the PC side). If the mobile phone side determines whether it is in the secure input interface, if not, the default window display is used. If so, the task ID and cover screen corresponding to the application in the secure input interface are sent to the PC (the cover screen is optionally pre-stored on the PC). After the user enters the password (the flag corresponding to the window state is reset), the mobile phone notifies the PC. The PC cancels the cover of the application window corresponding to the secure input interface.

[0228] It is understandable that, compared to traditional solutions, when a security flag is encountered, the content on the display will not be projected to the peer device. The solution of the embodiment of the present application will project the content on the display to the peer device regardless of whether the security flag is present or not, and at the same time transmit the security flag, telling the PC to cover the security flag with a security layer, thereby achieving security protection for multi-screen collaboration and multi-window.

[0229] In summary, in the display method of the embodiment of the present application, the first device determines whether each window interface has security requirements before screen projection, and then, based on the detection results, the first device sends the taskID of each window interface and the projection information of the window interface to the second device. The second device displays the security interface and the projection interface containing the window interface content based on the taskID, wherein the security interface is the projection interface of the window interface with security requirements. In this way, the second device can not only safely display the interface with security requirements, but also ensure that the user can operate normally on the interface without security requirements.

[0230] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0231] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A display method, characterized in that: The method comprises: The first device receives a screen projection instruction input by a user; In response to the screen projection instruction, the first device projects the first window interface to the second device; The first device runs the second window interface in response to the instruction; The first device determines that the second window interface includes a security flag bit; The first device sends the task number corresponding to the second window interface and the security protection instruction to the second device, and the security protection instruction is used to instruct the second device to display the projection interface of the second window interface as a security interface.

2. The method according to claim 1, wherein The first device responds to the instruction to run the second window interface, including: The first device receives a first operation instruction from the second device, the first operation instruction is associated with a first application APP, and the second window interface is a window interface of the first APP; The first device runs the first APP to start the second window interface.

3. The method according to claim 1, wherein The first device responds to the instruction to run the second window interface, including: The first device receives a second operation instruction input by a user, where the second operation instruction corresponds to a first application program APP, and the second window interface is a window interface of the first APP; The first device runs the first APP and displays the second window interface.

4. The method according to claim 1 or 2, wherein: After the first device sends the task number and the security protection instruction corresponding to the second window interface to the second device, the method further includes: The first device receives a third operation instruction from the second device, where the third operation instruction includes a task number corresponding to the second window interface; In response to the third operation instruction, the first device displays the second window interface; The first device receives operation information input by the user based on the second window interface; The first device updates the second window interface to a third window interface; The first device projects the third window interface to the second device.

5. The method according to any one of claims 1 to 4, wherein The first device determines that the second window interface includes a security flag bit, including: The first device calls the window management service of the first device to traverse the window state of the second window interface; If the window status of the second window interface includes a security flag, the first device determines that the second window interface includes a security flag.

6. The method according to claim 5, wherein The security flag includes a first flag, a second flag and a third flag. The first flag indicates that a security lock is applied, the second flag indicates that the application is actively set, and the third flag indicates that a security keyboard is called.

7. The method according to claim 1, wherein The security interface includes first prompt information, and the first prompt information is used to prompt the user to perform an operation on the first device side.

8. The method according to claim 1, wherein The security interface includes second prompt information, and the second prompt information is used to remind the user to pay attention to privacy protection.

9. The method according to claim 1, wherein The security interface includes third prompt information, and the third prompt information is used to prompt the user to display the application on the first device.

10. The method according to claim 9, wherein When the first device displays the security interface, the method further includes: When the user performs a first operation on the third prompt information, the first device receives information associated with the first operation, and the first device displays the second interface of the second application.

11. The method according to claim 10, wherein The information associated with the first operation includes an identifier corresponding to the second application.

12. The method according to any one of claims 1 to 11, wherein The method further comprises: The first device sends the first wallpaper of the first device to the second device, so that the second device displays an image associated with the first wallpaper on the security interface.

13. The method according to claim 12, wherein: The method further comprises: The image associated with the first wallpaper is an image of the first wallpaper after Gaussian blur processing.

14. The method according to claim 12, wherein: The method further comprises: The first device switches to use the second wallpaper, and the first device sends the second wallpaper to the second device, so that the second device presents an image associated with the second wallpaper on the security interface.

15. An electronic device, characterized in that: Used as a first device, the electronic device includes a memory and one or more processors; wherein the memory is used to store computer program code, and the computer program code includes computer instructions; when the computer instructions are executed by the processor, the electronic device executes the display method as described in any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that The present invention comprises a computer program, which, when executed on a computer, causes the computer to execute the display method according to any one of claims 1 to 14.

Images