A screen projection method, system and related device

Abstract

This application discloses a screen mirroring method, system, and related apparatus. The method includes: a first electronic device sending screen mirroring data to a second electronic device. If the first screen mirroring window in the screen mirroring data has changed compared to the previous frame of screen mirroring data, but the second screen mirroring window has not changed compared to the previous frame of screen mirroring data, then the second electronic device can render a new screen mirroring page in the first screen mirroring window and continue using the screen mirroring page already rendered in the previous frame in the second screen mirroring window. This reduces the workload of the GPU in the second electronic device and improves screen mirroring performance.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a screen projection method, system and related device. Background Technology

[0002] With the development of digital technology, the transformation of mobile phones and other electronic devices into personal computers (PCs) is one of the important means of realizing mobile office work. Specifically, after establishing a connection with a PC, a mobile phone can project multiple interfaces from the phone onto the PC in real time. Simultaneously, mobile phones and PCs can seamlessly share files such as pictures and videos, and users can use a mouse and keyboard on the PC to quickly process mobile messages and files.

[0003] When projecting an application interface from a mobile phone to a PC, the phone maintains its resolution and encodes and transmits the image data. Upon receiving the image data, the PC can display multiple windows. Since these multiple windows are rendered at the same frame rate, the graphics processing unit (GPU) inevitably becomes overloaded. This ultimately leads to display lag and stuttering on the PC. Summary of the Invention

[0004] This application provides a screen projection method, system, and related apparatus, which enables the large-screen device to continue using the rendered screen projection image from the previous frame if the projected image has not changed compared to the previous frame when the electronic device projects the image to the large-screen device. This reduces the workload of the GPU of the large-screen device and improves the screen projection performance.

[0005] In a first aspect, this application provides a screen mirroring system, comprising: a first electronic device and a second electronic device; wherein, the first electronic device is configured to acquire first display data corresponding to a first screen mirroring window on the second electronic device and second display data corresponding to a second screen mirroring window on the second electronic device; the first electronic device is further configured to send the first display data and the second display data to the second electronic device; the second electronic device is further configured to render and display a first screen mirroring page in the first screen mirroring window based on the first display data, and render and display a second screen mirroring page in the second screen mirroring window based on the second display data; the first electronic device is further configured to acquire third display data corresponding to the first screen mirroring window on the second electronic device and fourth display data corresponding to the second screen mirroring window on the second electronic device; the first electronic device is further configured to send the third display data and the fourth display data to the second electronic device; the second electronic device is further configured to, when it is determined that the third display data is different from the first display data and the fourth display data is the same as the second display data, render and display a third screen mirroring page in the first screen mirroring window based on the third display data, and display the cached second screen mirroring page in the second screen mirroring window.

[0006] This application provides a screen mirroring method in which a first electronic device can send screen mirroring data to a second electronic device. If the first screen mirroring window in the screen mirroring data has changed compared to the previous frame of screen mirroring data, but the second screen mirroring window has not changed compared to the previous frame of screen mirroring data, then the second electronic device can render a new screen mirroring page in the first screen mirroring window and continue using the screen mirroring page already rendered in the previous frame in the second screen mirroring window. This reduces the workload of the GPU in the second electronic device and improves screen mirroring performance.

[0007] In one possible implementation, sending the first display data and the second display data to the second electronic device specifically includes: drawing the first display data and the second display data in a first projection image frame, and recording the area range of the first display data and the area range of the second display data in the first projection image frame; and sending the first projection image frame, the area range of the first display data in the first projection image frame, and the area range of the second display data in the first projection image frame to the second electronic device. The second electronic device is further configured to: before rendering and displaying a first projection page in the first projection window based on the first display data, and before rendering and displaying a second projection page in the second projection window based on the second display data, crop the first display data from the first projection image frame based on the area range of the first display data in the first projection image frame, and crop the second display data from the second projection image frame based on the area range of the second display data in the first projection image frame.

[0008] In this way, the display data of the first projection window and the display data of the second projection window can be combined into a single frame for transmission, avoiding delays in the projection screen caused by the display data of the first projection window and the second projection window not arriving at the same time.

[0009] In one possible implementation, sending the third display data and the fourth display data to the second electronic device specifically includes: drawing the third display data and the fourth display data in a second projection image frame, and recording the area range of the third display data and the area range of the fourth display data in the second projection image frame; and sending the first projection image frame, the area range of the first display data in the first projection image frame, and the area range of the second display data in the first projection image frame to the second electronic device. The second electronic device is further configured to: before rendering and displaying a third projection page in the first projection window based on the first display data, and before displaying the cached second projection page in the second projection window, crop the third display data from the second projection image frame based on the area range of the third display data in the second projection image frame.

[0010] In this way, the display data of the first projection window and the display data of the second projection window can be combined into a single frame for transmission, avoiding delays in the projection screen caused by the display data of the first projection window and the second projection window not arriving at the same time.

[0011] In one possible implementation, the first electronic device is further configured to: determine the range of dirty regions in the second projected image frame where the image content differs from that of the first projected image frame; and send the dirty region range to the second electronic device. Determining that the third display data is different from the first display data and that the fourth display data is the same as the second display data specifically includes: when the dirty region range overlaps with the region range of the third display data in the second projected image frame, but does not overlap with the region range of the fourth display data in the second projected image frame, determining that the third display data is different from the first display data and that the fourth display data is the same as the second display data.

[0012] In this way, the first electronic device can mark the range of changes in the projected image frame and send the dirty area range to the second electronic device. When the second electronic device receives the projected image frame, it will cut the interface within the dirty area range and render it. For images that are not within the dirty area range, the second electronic device will continue to use the already rendered image from the previous frame, avoiding the situation of repeatedly rendering the same frame, reducing the workload of the second electronic device's GPU, and improving the projection performance.

[0013] In one possible implementation, the second electronic device is further configured to render and display a third projection page in the first projection window based on the third projection data, and to render and display a fourth projection page in the second projection window based on the third projection data, when it is determined that the third projection data is different from the first projection data and the fourth projection data is different from the second projection data.

[0014] In this way, when the images in both the first and second projection windows are updated, the second electronic device can re-render the image through the GPU in both the first and second projection windows.

[0015] In one possible implementation, the second electronic device is further configured to display the cached first projection page in the first projection window and the cached second projection page in the second projection window when it is determined that the third display data is the same as the first display data and the fourth display data is the same as the second display data.

[0016] In this way, when neither the first nor the second projection window is updated, the second electronic device can use the rendered previous frame to display in both the first and second projection windows.

[0017] In one possible implementation, the first electronic device is further configured to establish a Wi-Fi Direct connection with the second electronic device before acquiring the first display data corresponding to the first projection window on the second electronic device and the second display data corresponding to the second projection window on the second electronic device. Specifically, the first electronic device is configured to send the first display data and the second display data to the second electronic device via the Wi-Fi Direct connection. Specifically, the first electronic device is configured to send the third display data and the fourth display data to the third electronic device via the Wi-Fi Direct connection.

[0018] Secondly, this application provides a screen mirroring method applied to a second electronic device, characterized in that it includes: the second electronic device receiving first display data corresponding to a first screen mirroring window and second display data corresponding to a second screen mirroring window sent by a first electronic device; the second electronic device rendering and displaying a first screen mirroring page in the first screen mirroring window based on the first display data, and rendering and displaying a second screen mirroring page in the second screen mirroring window based on the second display data; the second electronic device receiving third display data corresponding to the first screen mirroring window and fourth display data corresponding to the second screen mirroring window sent by the first electronic device; when the second electronic device determines that the third display data is different from the first display data and the fourth display data is the same as the second display data, the second electronic device rendering and displaying a third screen mirroring page in the first screen mirroring window based on the third display data, and displaying the cached second screen mirroring page in the second screen mirroring window.

[0019] In one possible implementation, the second electronic device receives first display data corresponding to the first projection window and second display data corresponding to the second projection window sent by the first electronic device. Specifically, the second electronic device receives a first projection image frame, a region of the first display data in the first projection image frame, and a region of the second display data in the first projection image frame sent by the first electronic device. The first projection image frame includes the first display data and the second display data. The second electronic device crops the first display data from the first projection image frame based on the region of the first display data in the first projection image frame, and crops the second display data from the second projection image frame based on the region of the second display data in the first projection image frame.

[0020] In one possible implementation, the second electronic device receives third and fourth display data sent by the first electronic device, specifically including: the second electronic device receiving a second projection image frame, a region of the third display data in the second projection image frame, and a region of the second display data in the second projection image frame sent by the first electronic device; wherein, the second projection image frame includes the third and fourth display data; the second electronic device crops the third display data from the second projection image frame based on the region of the first display data in the second projection image frame, and crops the fourth display data from the second projection image frame based on the region of the fourth display data in the second projection image frame.

[0021] In one possible implementation, the method further includes: the second electronic device receiving a dirty region range in the second projection image frame sent by the first electronic device where the image content differs from the first projection image frame; the second electronic device determining that the third display data is different from the first display data and the fourth display data is the same as the second display data, specifically including: when the dirty region range overlaps with the region range of the third display data in the second projection image frame but does not overlap with the region range of the fourth display data in the second projection image frame, the second electronic device determines that the third display data is different from the first display data and the fourth display data is the same as the second display data.

[0022] In one possible implementation, the method further includes:

[0023] When the second electronic device determines that the third display data is different from the first display data and the fourth display data is different from the second display data, the second electronic device renders and displays the third projection page in the first projection window based on the third display data, and renders and displays the fourth projection page in the second projection window based on the third display data.

[0024] In one possible implementation, the method further includes: when the second electronic device determines that the third display data is the same as the first display data and the fourth display data is the same as the second display data, displaying the cached first projection page in the first projection window and displaying the cached second projection page in the second projection window.

[0025] In one possible implementation, before the second electronic device receives the first display data corresponding to the first projection window and the second display data corresponding to the second projection window sent by the first electronic device, the method further includes: the second electronic device establishing a Wi-Fi Direct connection with the first electronic device. The second electronic device receiving the first display data corresponding to the first projection window and the second display data corresponding to the second projection window sent by the first electronic device specifically includes: the second electronic device receiving the first display data and the second display data sent by the first electronic device via the Wi-Fi Direct connection. The second electronic device receiving the third display data corresponding to the first projection window and the fourth display data corresponding to the second projection window sent by the first electronic device specifically includes: the second electronic device receiving the third display data and the fourth display data sent by the first electronic device via the Wi-Fi Direct connection.

[0026] Thirdly, this application provides an electronic device, a second electronic device, which includes one or more processors, a Wi-Fi module, and one or more memories. The one or more memories and the Wi-Fi module are coupled to the one or more processors. The one or more memories are used to store computer program code, which includes computer instructions. When the one or more processors execute the computer instructions, they cause the communication device to perform the screen projection method in any of the possible implementations of any of the above aspects.

[0027] Fourthly, embodiments of this application provide a computer storage medium including computer instructions, which, when executed on a second electronic device, cause a communication device to perform the screen projection method in any of the possible implementations of any of the above aspects.

[0028] Fifthly, embodiments of this application provide a computer program product that, when run on a second electronic device, causes the second electronic device to execute the screen projection method in any of the possible implementations of any of the above aspects.

[0029] For the beneficial effects of aspects two through five, please refer to the beneficial effects of aspect one, which will not be repeated here. Attached Figure Description

[0030] Figure 1 A schematic diagram of a communication system provided in an embodiment of this application;

[0031] Figure 2A A schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of this application;

[0032] Figure 2B A software structure block diagram of an electronic device 100 provided in this application embodiment;

[0033] Figure 3 A schematic diagram of the hardware structure of an electronic device 200 provided in an embodiment of this application;

[0034] Figures 4A-4I A set of schematic diagrams of screen projection scenarios provided in the embodiments of this application;

[0035] Figure 5 A flowchart illustrating a screen mirroring method provided in an embodiment of this application;

[0036] Figure 6 This is a schematic diagram of the dirty area range provided in an embodiment of this application;

[0037] Figure 7 A flowchart is provided for an embodiment of this application;

[0038] Figures 8A-8BA set of schematic diagrams showing screen projection cut-off provided in the embodiments of this application;

[0039] Figure 9 A flowchart of an image encoding module provided in an embodiment of this application;

[0040] Figure 10 A flowchart of a screen projection method module provided in an embodiment of this application;

[0041] Figure 11 This is a flowchart illustrating a screen mirroring method provided in an embodiment of this application. Detailed Implementation

[0042] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0043] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0044] The following describes one method provided in this application for resolving screen stuttering during screen mirroring.

[0045] When mobile phones or other electronic devices detect lag in the screen projection from a large-screen device, they can reduce the resolution of the projected image data during encoding. After the mobile phone or other electronic device sends the projection data to the large-screen device, the large-screen device can then proportionally stretch and display the projected image.

[0046] However, due to the reduced resolution of the projected image, the image displayed on the large screen device will be blurry and unclear.

[0047] The following describes another method provided in this application to solve the problem of screen stuttering during screen mirroring.

[0048] When mobile devices detect lag in the screen projection from a large screen, they can maintain the resolution of the focused window (the currently active window) while reducing the resolution of non-focused windows during the encoding of the projection data. After the mobile device sends the projection data to the large screen, the large screen can then proportionally stretch and display the projected image.

[0049] However, due to the reduced resolution of the non-focus window's projection page, some parts of the image displayed on the large screen device will become blurry and unclear.

[0050] Therefore, this application provides a screen projection method in which the electronic device can mark the range of change of the projected image frame (this range of change can be called the dirty region range, or simply the dirty region range, dirty rectangle, etc., which is the range of change of one frame of image relative to the previous frame). The electronic device can send the dirty region range to the large-screen device. When the large-screen device receives the projection data, it will cut the interface of the dirty region range and render and display it. For images that are not in the dirty region range, the large-screen device will continue to use the already rendered image of the previous frame, avoiding the situation of repeatedly rendering the same frame, reducing the workload of the GPU of the large-screen device, and improving the projection performance.

[0051] The following describes a communication system provided in an embodiment of this application.

[0052] Figure 1 A schematic diagram of a communication system provided in an embodiment of this application is shown.

[0053] like Figure 1 As shown, the communication system 10 may include an electronic device 100 (also referred to as a first electronic device) and an electronic device 200 (also referred to as a second electronic device). Electronic device 100 can establish a wireless connection with electronic device 200 via wireless communication methods (e.g., Wireless Fidelity (Wi-Fi), Bluetooth, etc.). Electronic device 100 can wirelessly transmit file data to electronic device 200, or electronic device 100 can project an application interface onto electronic device 200 for display. In the communication system 10, the transmission of real-time data can be controlled using the Real-Time Streaming Protocol (RTSP). RTSP is a multimedia streaming protocol used to control audio or video, allowing simultaneous control of multiple streaming requests. Electronic device 100 can control the transmission of data streams via RTSP.

[0054] For example, when using Wi-Fi peer-to-peer (P2P) technology for screen mirroring, electronic device 100 can mix compressed H.264 format video or advanced audio coding (AAC) format audio into a transport stream (TS) file, and send the TS file to electronic device 200 via Wi-Fi P2P using the RTSP protocol. Electronic device 200 receives the TS file from electronic device 100 via the RTSP protocol and decodes and plays it. Here, H.264 is a video codec protocol, and AAC is an audio codec protocol.

[0055] In this embodiment of the application, the electronic device 100 may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device and / or smart city device, etc.

[0056] Electronic device 200 can be a television, desktop computer, laptop computer, handheld computer, notebook computer, super mobile personal computer, netbook, augmented reality device, virtual reality device, artificial intelligence device, in-vehicle device, smart home device, etc.

[0057] In one possible implementation, electronic device 100 can also be connected to electronic device 200 via an adapter. The adapter includes a Universal Serial Bus (USB) interface for connecting data cables and a Type-C interface for data cables, allowing electronic device 100 to be connected to the adapter via a data cable. The adapter can also be connected to electronic device 200 via an adapter cable, which can include any of the following: High Definition Multimedia Interface (HDMI) cable, Video Graphics Array (AGI) cable, Digital Visual Interface (DVI) cable, etc. In this embodiment, the adapter can be a Mobile High Definition Link (MHL) device, which can simultaneously transmit audio or video signals from electronic device 100 (such as a mobile phone) to electronic device 200 for display. Electronic device 100 can act as a master device, and electronic device 200 can act as a slave device of electronic device 100. After electronic device 100 and electronic device 200 establish a connection, electronic device 100 essentially has two displays: one is its own touch screen, and the other is the display of electronic device 200 connected to it.

[0058] The following is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0059] Figure 2A A schematic diagram of the hardware structure of the electronic device 100 is shown.

[0060] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0061] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0062] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processors, image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0063] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0064] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0065] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I1C) interface, an inter-integrated circuit sound (I1S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0066] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0067] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0068] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0069] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0070] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0071] The mobile communication module 150 can provide solutions for wireless communication, including 1G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0072] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs an audio signal through an audio device (not limited to speaker 170A, receiver 170B, etc.), or projects a page or video onto the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0073] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0074] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0075] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0076] The display screen 194 is used for projecting web pages, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0077] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0078] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0079] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0080] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0081] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG1, MPEG3, MPEG4, etc.

[0082] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0083] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0084] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0085] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0086] The audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal.

[0087] The loudspeaker 170A, also known as a "loudspeaker", is used to convert audio electrical signals into sound signals.

[0088] The receiver 170B, also known as the "earpiece", is used to convert audio electrical signals into sound signals.

[0089] The microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals.

[0090] When pressure sensor 180A is used to sense pressure signals, it can convert the pressure signals into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. Gyroscope sensor 180B can be used to determine the motion posture of electronic device 100. Barometric pressure sensor 180C is used to measure barometric pressure. Magnetic sensor 180D includes a Hall sensor and can be used to detect the opening and closing of the flip cover. Accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in various directions (generally three axes). Distance sensor 180F is used to measure distance. Proximity sensor 180G can also be used for automatic unlocking and locking in flip cover mode and pocket mode. Ambient light sensor 180L is used to sense ambient light brightness. Fingerprint sensor 180H is used to collect fingerprints. Temperature sensor 180J is used to detect temperature. Touch sensor 180K, also called "touch panel". Touch sensor 180K can be disposed on display screen 194, and touch sensor 180K and display screen 194 constitute a touch screen, also called "touch screen". A touch sensor 180K is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided via the display screen 194. In some embodiments, the touch sensor 180K may also be located on the surface of the electronic device 100, in a different position than the display screen 194. A bone conduction sensor 180M can acquire vibration signals. Buttons 190 include a power button, volume buttons, etc. A motor 191 can generate vibration cues. An indicator 192 may be an indicator light, used to indicate charging status, battery level changes, or messages, missed calls, notifications, etc. A SIM card interface 195 is used to connect a SIM card.

[0091] The following is a software structure diagram of the electronic device 100 provided in the embodiments of the present invention.

[0092] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0093] Figure 2B This is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention.

[0094] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0095] The application layer can include a series of application packages.

[0096] like Figure 2B As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and screen mirroring.

[0097] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0098] like Figure 2B As shown, the application framework layer may include a window manager, content provider, view system, resource manager, image compositor API, multimedia encoder API, etc.

[0099] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0100] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0101] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0102] The image compositer API can call relevant service processes to composite UI data from various layers into a single layer, which can be used in conjunction with the encoder. The image composite module API can also detect whether the projected interface has changed.

[0103] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0104] The Multimedia Encoder API can call relevant service processes to create encoders / decoders for specific file types (such as audio / video file types like "video / avc"). For example, a multimedia encoder can encode and decode image data synthesized by a virtual display module based on the H.264 video codec protocol.

[0105] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0106] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0107] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0108] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0109] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0110] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0111] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0112] A 2D graphics engine is a graphics engine for 2D drawing.

[0113] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0114] The following is a schematic diagram of the hardware structure of an electronic device 200 provided in an embodiment of this application.

[0115] like Figure 3 As shown, the electronic device 200 may include: a processor 222, a memory 223, a wireless communication module 224, a power switch 225, a display screen 229, and an audio module 230. In some embodiments, the electronic device 200 may further include a wired LAN communication processing module 226, a high-definition multimedia interface (HDMI) communication processing module 227, a USB communication processing module 228, etc. These modules can be connected via a bus.

[0116] Processor 222 can be used to read and execute computer-readable instructions. In a specific implementation, processor 222 mainly includes a controller, an arithmetic logic unit (ALU), and registers. The controller is primarily responsible for instruction decoding and issuing control signals for the operations corresponding to the instructions. The ALU is primarily responsible for performing fixed-point or floating-point arithmetic operations, shift operations, and logical operations, and can also perform address operations and translations. Registers are primarily responsible for storing register operands and intermediate operation results temporarily stored during instruction execution. In a specific implementation, the hardware architecture of processor 222 can be an application-specific integrated circuit (ASIC) architecture, a MIPS architecture, an ARM architecture, or an NP architecture, etc.

[0117] In this embodiment, the processor 222 can be used to parse signals received by the wireless communication module 224, such as a new URL sent by the electronic device 100, and obtain multiple videos and associated videos in the playlist based on the new URL.

[0118] The wireless communication module 224 may include a WLAN communication processing module. Optionally, the wireless communication module 224 may also include a Bluetooth (BT) communication processing module, an NFC processing module, a cellular mobile communication processing module (not shown), etc.

[0119] In this embodiment, the wireless communication module 224 can be used to establish a communication connection with the electronic device 100. The communication connection established between the wireless communication module 224 and the electronic device 100 can be of various types. For example, the WLAN communication processing module can be used to establish a Wi-Fi Direct communication connection with the electronic device 100, the Bluetooth (BT) communication processing module can be used to establish a Bluetooth communication connection with the electronic device 100, the NFC processing module can be used to establish an NFC connection with the electronic device 100, and so on.

[0120] In this embodiment, the wireless communication module 224 can also be used to establish a communication connection with the electronic device 100 and receive video streams sent by the electronic device 100 based on the communication connection. The communication connection established between the wireless communication module 224 and the electronic device 100 can transmit data based on the HTTP protocol. This application does not impose any restrictions on the type of communication connection or data transmission protocol between devices.

[0121] Memory 223 is coupled to processor 222 and is used to store various software programs and / or sets of instructions. In specific implementations, memory 223 may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 223 may store an operating system, such as uCOS, VxWorks, RTLinux, or other embedded operating systems. Memory 223 may also store communication programs that can be used to communicate with electronic device 100, one or more servers, or additional devices.

[0122] The power switch 225 can be used to control the power supply to the electronic device 200.

[0123] The wired LAN communication processing module 226 can be used to communicate with other devices in the same LAN via a wired LAN, and can also be used to connect to a WAN via a wired LAN to communicate with devices in the WAN.

[0124] The HDMI communication processing module 227 can be used to communicate with other devices via an HDMI interface (not shown).

[0125] The USB communication processing module 228 can be used to communicate with other devices via a USB interface (not shown).

[0126] The display screen 229 can be used to project web pages, videos, etc. The display screen 229 can be an LCD, OLED, AMOLED, FLED, QLED, or other display technologies. The content displayed on the display screen 229 can be found in the relevant descriptions of the subsequent method embodiments.

[0127] In this embodiment, the display screen 229 can continuously play multiple videos based on the video streams of multiple videos, such as playlists and associated videos, sent by the server 300 and received by the wireless communication module 224.

[0128] Audio module 230. Audio module 230 can be used to output audio signals through an audio output interface, thus enabling the large-screen display device 200 to support audio playback. Audio module 230 can also be used to receive audio data through an audio input interface. Audio module 230 may include, but is not limited to, a microphone, speaker, receiver, etc.

[0129] In some embodiments, the electronic device 200 may also include a serial interface such as an RS-232 interface. This serial interface can be connected to other devices, such as speakers or other audio playback devices, enabling the display and the audio playback device to collaboratively play audio and video.

[0130] Understandable Figure 3 The illustrated structure does not constitute a specific limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0131] In some embodiments, the electronic device 200 may include the above-described components. Figure 1 The electronic device 100 shown includes hardware. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0132] The software system of electronic device 200 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc. For example, the software system of electronic device 200 may include, but is not limited to, [other types of architectures]. Linux or other operating systems. It is Huawei's HarmonyOS operating system.

[0133] Figures 4A-4I The diagram shows a set of interface schematics for screen projection scenarios provided in the embodiments of this application.

[0134] In some application scenarios, after electronic device 100 and electronic device 200 establish their first connection, electronic device 100 can combine multiple application interfaces into a single projection image frame. Electronic device 100 can also identify areas in the projection image frame that have changed compared to the previous frame (dirty area range). Electronic device 100 can send the projection image frame and dirty area range data to electronic device 200, which can then process and display the projection image frame. Simultaneously, electronic device 100 and electronic device 200 can seamlessly share files such as images or videos. Users can also quickly process messages and files on electronic device 100 via an input device on electronic device 200. First, electronic device 200 can display any interface (e.g., desktop). Upon receiving a projection request from electronic device 100, electronic device 200 can display a projection window above the current interface, showing the application interface projected by electronic device 100. After receiving and agreeing to the projection request from electronic device 100, electronic device 200 can receive and display the application interface from electronic device 100.

[0135] For example, such as Figure 4A As shown, the electronic device 100 can display a drop-down menu window 410. This drop-down menu window 410 may include a wireless projection control 411 and other function switch controls (e.g., WLAN switch control, Bluetooth control, airplane mode switch control, auto-rotate switch control, flashlight switch control, etc.). The wireless projection control 411 can be used to trigger the electronic device 100 to establish a WLAN direct connection with other devices and send the display data on the electronic device 100 to other devices for display. The electronic device 200 can display a user interface 400, which may display icons including, but not limited to, "This PC".

[0136] Electronic device 100 can receive user input (e.g., a click) to the wireless projection control 411, and in response to this input, the display of electronic device 100 can show, for example... Figure 4B The device selection interface shown is 420.

[0137] like Figure 4B As shown, the device selection interface 420 may include one or more device options, among which is a device option 421 corresponding to the electronic device 200. The device name of the electronic device 200 may be "HUAWEI PC", and the device option 421 may display the device name of the electronic device 200 (e.g., "HUAWEIPC").

[0138] Electronic device 100 can receive user input (e.g., a click) for device option 421. In response to this input, electronic device 100 can send a screen mirroring connection request to electronic device 200. Upon receiving the screen mirroring connection request, electronic device 200 can display, as shown below... Figure 4C The screen mirroring prompt window 431 is shown.

[0139] like Figure 4C As shown, the prompt window 431 displayed on the electronic device 200 may include a confirmation control 432, a rejection control 433, and a text prompt 434. The text prompt 434 can be used to prompt the user on the electronic device 200 to confirm whether to accept the screen mirroring request from the electronic device 100. For example, the text prompt could be "Electronic device HUAWEIPHONE requests to establish a screen mirroring connection, do you agree?". The confirmation control 432 can be used to trigger the electronic device 200 to accept the screen mirroring connection request from the electronic device 100. The rejection control 433 can be used to trigger the electronic device 200 to reject the screen mirroring connection request from the electronic device 100.

[0140] Electronic device 200 can receive user input (e.g., a click) applied to the confirmation control 432. In response to this input, electronic device 200 can establish a WLAN direct connection with electronic device 100. After establishing a WLAN direct connection, electronic device 100 can project the image displayed on its screen to the screen of electronic device 200 in real time via the WLAN direct connection. Specifically, electronic device 100 can send projection data, including projection image frame 1 and dirty area range information 1, to electronic device 200. Electronic device 200 can then crop, render, and display the image within the dirty area range in the projection window.

[0141] For example, such as Figure 4D As shown, the device selection interface 420 is currently displayed on the screen of electronic device 100. Electronic device 100 can project this device selection interface 420 onto electronic device 200. Electronic device 200 can display a projection window 451 and a projection window function menu 456. The projection window function menu 456 is the function menu of projection window 451. This projection window function menu 456 includes a minimize window control 453, a maximize window control 454, and a close window control 455. The minimize window control 453 can be used to minimize the projection window 451. The maximize window control 454 can be used to maximize the projection window 451. The close window control 455 can be used to close the projection window 451. Each projection window can correspond to one projection window function menu. Projection window 451 can display the projection page 452.

[0142] The content of the projection page 452 is the same as the content of the device selection interface 42 of the electronic device 100. The electronic device 200 can receive user input for the projection page 452 and send the input information (including the user's input location and input action information) to the electronic device 100. The electronic device 100 can respond accordingly and update the device selection interface 420 based on the received input information. The electronic device 100 can also send the updated device selection interface 420 to the electronic device 200. After receiving the screen sent by the electronic device 100, the electronic device 200 can display the updated device selection interface 420 in the projection window 451.

[0143] Electronic device 100 can receive user input (e.g., swiping up), and in response to that input, electronic device 100 can display, for example... Figure 4E The user interface 460 is shown. Electronic device 100 can send projection data, including projection image frame 2 and dirty area range information 2, to electronic device 200, which can crop, render, and display the image within the dirty area range in the projection window.

[0144] For example, such as Figure 4E As shown, the user interface 460 may include a memo icon 461, a video icon 462, a contacts icon, and a message icon. The memo icon 461 can be used to open the "Memo" application and display its interface. The video icon 462 can be used to open the "Video" application and display its interface. The user interface 460 may also include a page indicator; application icons can be distributed across multiple pages, and the page indicator can be used to show which page the user is currently viewing. The user can swipe left or right on an area of ​​the user interface 460 to browse other pages. The user interface 460 can be the main interface. The electronic device 100 can project this user interface 460 onto the electronic device 200 for display.

[0145] The electronic device 200 can display a projection window 451, which can display a projection page 471. The content of the projection page 471 is the same as the content of the user interface 460 of the electronic device 100. The projection page 471 includes a memo icon 473, a video icon 474, etc. The memo icon 473 is similar to the memo icon 461. The video icon 474 is similar to the video icon 462.

[0146] Electronic device 200 can receive user input (e.g., long press) on video icon 474. In response to this input, electronic device 200 can send the input information corresponding to the input operation (including the location and type of the input operation) to electronic device 100. Electronic device 100 can then display a menu window 463 in response to this input information. Afterwards, electronic device 100 can send data including the projected image frame 3 and dirty area range information 3 to electronic device 200. Electronic device 200 can then crop, render, and display the image within the dirty area range in the corresponding projected window. Electronic device 200 can display... Figure 4F The screen mirroring page 472 is shown. The content of the screen mirroring page 472 is the same as the content of the user interface 460 currently displayed on the electronic device 100.

[0147] like Figure 4F As shown, the screen mirroring page 472 includes a memo icon 473, a menu window 475, etc. The menu window 475 may include, but is not limited to, an option to open in a new window 476, a search option for the entire web, and a popular viewing option.

[0148] Electronic device 200 can receive user input (e.g., a click) for the new window opening option 476, and respond to this input. Electronic device 200 can first send this input information to electronic device 100, and electronic device 100, in response to this input, can combine the video application interface and the main interface into a single projected image frame 4. Electronic device 100 can send data including the projected image frame 4 and dirty area range information 4 to electronic device 200, and electronic device 200 can crop, render, and display the image within the dirty area range in the corresponding projected window. (For example...) Figure 4G The screen mirroring pages 482 and 473 are shown. The electronic device 200 can also discard the portion of the screen outside the dirty area and display the previous frame in the corresponding screen mirroring window. For a description of the dirty area range, please refer to subsequent embodiments; it will not be repeated here.

[0149] like Figure 4G As shown, the electronic device 200 displays a user interface 400, which includes a projection window 451 and a projection window 481. Projection window 451 displays a projection page 473, the content of which is the same as the content of the user interface 460 currently displayed by the electronic device 100. Projection window 481 displays a projection page 482. Projection page 482 is a real-time video application interface sent by the electronic device 100. Projection page 482 includes a playback window 483, which is playing a video.

[0150] Electronic device 200 can receive user input operations (such as dragging a memo icon). In response to this input operation, electronic device 200 can first send the input information corresponding to the input operation to electronic device 100. In response to this input information, electronic device 100 can combine the memo application interface, video application interface, and main interface into a single projection image frame A (e.g., Figure 8A The screen projection image frame 800 is shown. Electronic device 100 can also mark the dirty region range of the screen projection image frame A to obtain dirty region range information A. Electronic device 100 can also encapsulate the dirty region range information A based on the RTSP protocol. Then, electronic device 100 can send the projection data A to electronic device 200. The projection data A includes the screen projection image frame A, the position information of multiple projection pages in the screen projection image frame A, and the dirty region range information A. After receiving the projection data A sent by electronic device 100, electronic device 200 can display the data within the dirty region range of the screen projection image frame A (such as...). Figure 8A The displayed data (801, 802, and 803) are cropped and rendered in the corresponding projection window (e.g., ...). Figure 4H Screen mirroring pages 484, 492, and 474 are shown.

[0151] like Figure 4H As shown, the electronic device 200 displays a user interface 400, which includes projection windows 451, 481, and 491. Projection window 451 displays projection page 474. Projection page 474, compared to projection page 473, also displays a pop-up window 475. Projection window 481 displays projection page 484. Projection page 484 is a real-time video application interface sent by the electronic device 100. Projection page 484 may include a playback window 485, which is playing a video. Projection window 491 displays projection page 492. Projection page 492 is a real-time memo application interface sent by the electronic device 100. Projection page 492 may include a new note control 495, which can be used to create a new note.

[0152] Electronic device 200 can receive user input (e.g., click) on newly created control 495 and respond to the input. Specifically, electronic device 200 can first send the input information corresponding to the input operation to electronic device 100. Electronic device 100, in response to this input information, can combine the memo application interface, video application interface, and main interface into a single projection image frame B (e.g., ...). Figure 8B(See screen projection image frame 810). Electronic device 100 can also determine the dirty region range in the screen projection image frame B, obtaining dirty region range information B. Electronic device 100 can send screen projection data B to electronic device 200. The screen projection data B includes the screen projection image frame B, the position information of multiple screen projection pages in the screen projection image frame B, and the dirty region range information B. After receiving the screen projection data B sent by electronic device 100, electronic device 200 can crop out the screen projection pages within the dirty region range from the screen projection image frame and display them in the corresponding screen projection window.

[0153] The electronic device 200 can also discard the screen projection page outside the dirty area range in the screen projection image frame and display the previous screen projection page in the screen projection window corresponding to the screen projection page.

[0154] like Figure 4I As shown, after receiving the projection data B sent by the electronic device 100, the electronic device 200 can obtain the projection image frame B, the position information of the display data corresponding to multiple projection windows, and the dirty area range information B from the projection data B. Based on the projection image frame B, the position information of the display data, and the dirty area range information B, the electronic device 200 can crop out the images within the dirty area from the projection image frame B. Figure 8B Display data 811 and display data 812 are shown. Electronic device 200 renders display data 811 into a projection page 493 and displays projection page 493 in projection window 491. Electronic device 200 can also render display data 812 into a projection page 486 and display projection page 486 in projection window 481. Since display data 813 in projection image frame B has not changed compared to display data 803, it is outside the dirty area. Electronic device 200 can continue to display the projection page 474 of the previous frame in projection window 451. Projection page 486 is a real-time view of the video application interface sent by electronic device 100. Projection page 486 may include a playback window 487, which is playing video. Projection window 491 displays projection page 493. Projection page 493 is a real-time view of the memo application interface sent by electronic device 100. This projection page 493 is a note editing page.

[0155] According to the screen casting method provided in this embodiment, electronic device 100 can synthesize a new screen casting image and send it to electronic device 200 after a predetermined time (e.g., 5ms). When the display content of a specified screen casting window changes (e.g., when the content displayed in the specified screen casting window includes a graphics-interchange format (GIF) animation or a playing video, the display content of the screen casting window will change; or when some functional controls of the specified screen casting window are selected by the user, the display content of the screen casting window will also change), the display content of the other screen casting windows remains unchanged. However, if electronic device 200 renders and displays the image of every screen casting window regardless of whether the display content of a particular screen casting window changes, it will waste GPU resources. Therefore, electronic device 100 can send data such as the dirty area range and screen casting image frames to electronic device 200. Electronic device 200 can parse the dirty area range and update the screen casting page of the screen casting window within the dirty area range in real time. Since electronic device 200 only updates the image within the dirty area range, it can effectively reduce GPU usage and reduce the probability of screen stuttering during screen casting.

[0156] Figure 5 A flowchart illustrating a screen projection method provided in an embodiment of this application is shown. This flowchart can be applied to the communication system 10 described above.

[0157] like Figure 5 As shown, a screen mirroring method provided in this application embodiment may include the following steps:

[0158] 1. Electronic device 100 can acquire display data from one or more screen-projected windows and write the display data to a designated storage area. For example, ... Figure 4I As shown, the electronic device 100 acquires the display data (main interface data) of the projection window 451, the display data (video application interface data) of the projection window 481, and the display data (memo application interface data) of the projection window 491, and writes the display data of the three projection windows into the designated storage area of ​​the electronic device 100.

[0159] 2. Electronic device 100 can synthesize the display data of one or more projection windows into a single projection image frame. During this process, electronic device 100 retrieves the relevant display data from a designated storage area, synthesizes it into a single projection image frame, writes the projection image frame into the designated storage area, and records the position information of the projection page corresponding to one or more projection windows. Simultaneously, electronic device 100 can obtain the dirty region range of the projection image frame. For example, electronic device 100 can add a dirty region range attribute to the shared attribute area. When synthesizing the projection image frame, electronic device 100 can compare it with the previous projection image frame to obtain the area where the image has changed. The obtained area where the image has changed is then written into the dirty region range attribute of the shared attribute area.

[0160] For example, electronic device 100 generates such as Figure 8A After the screen projection image frame 800 shown, the electronic device 100 can record the position information of the three screen projection pages in the screen projection image frame. The electronic device 100 can record the distances between the four sides of the displayed data 801 and the preset coordinate axes (the xy plane coordinate axes set by the electronic device 100 in the screen projection image frame). For example, the electronic device 100 can record the position information of the displayed data 801 as (20 pixel value, 180 pixel value, 15 pixel value, 70 pixel value), which represent the distances between the bottom edge of the displayed data 801 and the x-axis, the top edge of the displayed data 801 and the x-axis, the left edge of the displayed data 801 and the y-axis, and the right edge of the displayed data 801 and the y-axis, respectively.

[0161] in, Figure 6 A schematic diagram of the dirty region range provided in an embodiment of this application is shown.

[0162] For example, Figure 6 It can display Figure 4I The screen-projected image frame B obtained by the electronic device 100 shown is... Figure 8B The projected image frame 810 shown is relative to Figure 4H The screen-projected image frame A obtained by the electronic device 100 shown is... Figure 8A The areas in the projected image frame 800 that have changed are: Display data 811, which differs from display data 801, and can be marked as dirty region 601. Display data 812, which differs from display data 802 (the content of the playback window has changed), can also be marked as dirty region 602. Display data 813 is the same as display data 803 and does not need to be marked as a dirty region. Figure 6The non-dirty region 603 is shown. In this embodiment, when a portion of a page is within a dirty region, the page is marked as being within a dirty region. Therefore, the electronic device 100 can obtain the dirty region range information B of the projected image frame B. The dirty region range information B can indicate that display data 811 and display data 812 are within the dirty region, while display data 813 is not within the dirty region.

[0163] The process of synthesizing screen projection image frames by electronic device 100 will be described next.

[0164] This application provides a method for synthesizing projected image frames. The application of the electronic device 100 can call the SurfaceFlinger service to synthesize the measured, laid out, and drawn screen (Surface) into a single projected image frame. The SurfaceFlinger system service manages the display screen and has the function of synthesizing Surfaces, capable of combining 2D and 3D surfaces from various applications.

[0165] In this embodiment, a Surface can be understood as a window of an application. Each window of the application corresponds to a canvas, i.e., a Surface. Every change to the application window is accompanied by an update of the Surface's content. The electronic device 100 can generate a cache list, and can write updated display data from the application into the cache list. Updating the Surface content involves the generation and consumption of the cache list.

[0166] like Figure 7 As shown, during the processing of display data, the application process acts as a producer, generating display data for the application. Then, the electronic device 100 writes the application's display content into shared display memory. Afterwards, the SurfaceFlinger system service, acting as a consumer, retrieves the display data from the shared display memory and synthesizes it into a single frame for projection. This producer-consumer process occurs cyclically until the producer stops synthesizing data, at which point the consumer retrieves all the display data from the shared display memory.

[0167] Subsequently, the SurfaceFlinger system service of electronic device 100 can act as a producer, calling this service to write the projected image frame into a cache. Electronic device 100 can then add this cached data to a cache queue (this operation can be called enqueueing). The multimedia encoder of electronic device 100 acts as a consumer, using the multimedia encoder to access the cache list and write the projected data into the cache. After the multimedia encoder retrieves the data from the cache and encodes it, electronic device 100 can release the cached data and then remove the empty cache from the cache queue (this operation can be called dequeueing). Afterward, electronic device 100 can again write the projected image frame into an empty cache and then add this cached data to the cache queue. Figure 7 The process shown will occur alternately and cyclically until the producer stops working and the consumer releases all cache.

[0168] When the electronic device writes the data of the projected image frame into the cache queue, the electronic device 100 can compare the displayed content of each application with the previous frame to see if there have been any changes (for example, the electronic device 100 can compare two images pixel by pixel from top to bottom to obtain the range of image changes). The electronic device 100 can then write the obtained dirty region range into the shared attribute area of ​​the cache list. This process can be divided into the following steps:

[0169] (1) Expand the attribute fields of the cache list to add a dirty region range (dirty_region) attribute. This operation can be achieved through the following code:

[0170]

[0171]

[0172] The `attr_region` structure listed above lists the attribute fields of the cache list. Among them, `crc_src_rect` and `crc_vals` can be used to store checksums. `write_magic` and `write_padding` can be used to set the image borders. The `crop_bottom`, `crop_left`, `crop_top`, and `crop_right` attributes can be used to mark the image position. `use_yuv_transform` can be used to specify the image encoding method, and so on.

[0173] Here, `dirty_region` is an array used to mark the range of dirty regions. This is because the interfaces of multiple applications are arranged horizontally, for example... Figure 8AThe screen-cast image frame shown is 800. Therefore, it is only necessary to mark whether the interface area of ​​each application has changed. If it has changed (dirty area range), its value is assigned to 1; if it has not changed (non-dirty area range), its value is assigned to 0.

[0174] The operation of assigning a value to the dirty_region attribute can be implemented using the following code.

[0175]

[0176] The above program first obtains an array of dirty region ranges and assigns it to the `dirtyRegion` property. Then, it uses constant pointers to point to the beginning and end of the data stream to obtain the height of the projected image frame. Next, it initializes the `dirty_region` property to 0. Finally, it uses a `while` loop to traverse the data stream and obtain the dirty region range (also known as dirty region range information).

[0177] For example, in Figures 4H-4I In the scenario shown, firstly, electronic device 100 generates a frame as shown... Figure 8A The screen projection image frame 800 is shown below. Electronic device 100 first writes the memo application interface data, video application interface data, and main interface data to a designated storage area, and then calls the relevant process (e.g., SurfaceFlinger) to synthesize the display data of multiple applications into a single screen projection image frame 800. This screen projection image frame 800 includes display data 801, display data 802, and display data 803. Simultaneously, electronic device 100 can record the position information of the above three pages in the screen projection image frame. Afterwards, electronic device 100 can write the screen projection image frame 800 to a shared buffer so that the multimedia encoder can encode the projection data. Electronic device 100 can send the screen projection image frame 800 to electronic device 200 for cutting, rendering, and display. Afterwards, electronic device 100 generates a frame as shown below. Figure 8BThe projected image frame 810 is shown. This projected image frame 810 includes display data 811, display data 812, and display data 813. Electronic device 100 can compare projected image frame 810 with the previous projected image frame 800 to obtain the area (dirty region range) where projected image frame 810 has changed relative to the previous projected image frame 800. Electronic device 100 can determine that display data 811 and display data 812 are within the dirty region range, while display data 813 is not. Therefore, electronic device 100 can set the dirty_region value of display data 811 and display data 812 to 1, and the dirty_region value of display data 813 to 0. Then, electronic device 100 can send the dirty region range information, including the dirty region range attribute, along with projected image frame 810 to electronic device 200. Electronic device 200 can then crop, render, and display the display data 811 and display data 812.

[0178] 3. Electronic device 100 can use a multimedia encoder to encode the obtained projected image frames. Specifically, electronic device 100 controls the multimedia encoder to encode and decode the images using the H.264 video codec protocol. H.264 is also known as Advanced Video Coding (MPEG-4 AVC). Electronic device 100 can also use other video codec protocols. Optionally, electronic device 100 can capture and compress the audio being played by various applications, wherein the audio compression format can be Advanced Audio Coding.

[0179] Specifically, such as Figure 9 As shown, the Figure 9 A flowchart of the image encoding module provided in an embodiment of this application is shown. The specific steps are as follows.

[0180] S901, Multimedia Encoder: Creates an encoder and initializes relevant parameters.

[0181] Electronic device 100 can create an encoder using the `createEncoderByType` function. After creating the codec, electronic device 100 can set a callback method by calling the `setCallback` function interface. The electronic device 100 will execute the program callback operation after the encoder completes its encoding task. Afterwards, electronic device 100 can configure the codec using the specified media format and initialize related parameters (e.g., color format). The multimedia encoder of electronic device 100 can support encoding and decoding of various data types, such as compressed audio and video data, raw audio data, and raw video data. Electronic device 100 can query the input and output format results in the Configured state. This result can be used to verify the configuration before starting encoding and decoding.

[0182] S902, the multimedia encoder creates an input Surface instance.

[0183] After configuring the encoder, the electronic device 100 can call the createInputSurface function interface to create an input Surface instance, which can be used to input the screen projection image frame of the virtual display module.

[0184] S903, the virtual display module creates a virtual monitor instance.

[0185] Electronic device 100 calls the createVirtualDisplay function interface of the virtual display module to create a virtual display instance. The virtual display instance includes the address of the available cache configured by the system.

[0186] S904, the multimedia encoder calls the virtual display instance of the virtual display module.

[0187] The multimedia encoder obtains the address of the available buffer area by calling the virtual display instance of the virtual display module. The multimedia encoder can call the `setInputSurface` method to configure the buffer address of the encoder output for the created input Surface instance; this address is the address of the available buffer area provided by the virtual display instance. The virtual display module calls the image compositing module to synthesize the projected image frame and writes it to the buffer. The input Surface instance can provide the data from the buffer to the encoder.

[0188] S905, the multimedia encoder begins data encoding and writes dirty region range information.

[0189] When electronic device 100 calls the `start` method, the encoder can retrieve data from the buffer and encode it. When encoding H.264 projection interface data, the encoder retrieves a dirty region range from the shared memory area of ​​the specified buffer list, encapsulates the dirty region range according to the transport protocol, and writes it to the data stream file. Maintaining ownership of the input or output buffer allows the encoder to stop operating when it is not necessary to immediately resubmit or release the buffer. Specifically, the encoder may delay generating the output buffer until the output buffer is released or resubmitted.

[0190] S906, after the multimedia encoder finishes its work, the program callback occurs.

[0191] After the encoder finishes encoding, the electronic device 100 performs a program callback. Specifically, the electronic device 100 can return an end-of-stream flag via the callback method `onOutputBufferAvailable`, indicating that the input Surface has stopped submitting data to the encoder. This flag can be set in the last valid output buffer or by adding an empty buffer after the last valid output buffer.

[0192] S907, obtain the output result.

[0193] The encoder completes the encoding of the projected data, resulting in a data stream file. This data stream file includes projected image frames, position information, and dirty region range information encapsulated based on the RTSP protocol.

[0194] S908, data transmission.

[0195] Electronic device 100 transfers a data stream file to electronic device 200.

[0196] 4. Electronic device 100 generates a data stream file from the encoded image / audio.

[0197] The data stream file includes the projected image frame, the dirty area range information of the projected image frame, and the position information of the display data corresponding to the projected window in the projected image frame.

[0198] For example, electronic device 100 can encode the projected image frame 800, and then write the encoded projected image frame 800 into a data stream file.

[0199] 5. Electronic device 100 can use the real-time streaming protocol to send the above data stream file to electronic device 200 via WLAN direct connection.

[0200] 6. Electronic device 200 receives data stream files from electronic device 100 via a real-time streaming protocol.

[0201] 7. Electronic device 200 decodes the data stream file to obtain the projected image frame.

[0202] 8. Electronic device 200 can parse the protocol used for screen projection, obtain dirty area range information, and then segment the image of each page within the dirty area range according to the dirty area range information and the position and size of each page in the screen projection image frame.

[0203] For example, the projected image frame is as follows: Figure 8BThe electronic device 200 can obtain the range of the three projected pages in the projected image frame 810, and whether the projected page is within the dirty region range, based on the above steps. The dirty_region values ​​of display data 811 and 812 are 1, while the dirty_region value of display data 813 is 0. Therefore, the electronic device 200 can identify that display data 811 and 812 are within the dirty region range, and display data 813 is within the non-dirty region range. The electronic device 200 will then slice the images of display data 811 and 812 and write them to a designated storage area.

[0204] 9. Electronic device 200 sends the segmented image to the graphics processing unit (GPU). The GPU renders the dirty area portion of the page and sends it to its corresponding projection window for display. After discarding the non-dirty area portion of the page, electronic device 200 controls the projection window corresponding to that page to continue displaying the previous image.

[0205] For example, such as Figure 4I As shown, the electronic device 200 displays a user interface 400, which includes projection windows 451, 481, and 491. Projection window 451 displays projection page 473. Projection window 481 displays projection page 486. Projection window 491 displays projection page 493. The pages displayed in projection windows 481 and 491 are within the dirty region, so projection pages 486 and 493 are rendered and displayed by the electronic device 200. The page displayed in projection window 451 is not within the dirty region, so the electronic device 200 uses the previous frame's projection page 473.

[0206] In this way, since electronic device 100 identifies and marks the dirty area range, when electronic device 100 sends the screen projection data, it also sends the dirty area range attribute to electronic device 200. Electronic device 200 can then detect which screen projection windows have unchanged display data and can reuse the GPU-rendered image. Electronic device 200 can save GPU computing time and avoid screen stuttering during projection.

[0207] The following is a flowchart illustrating the internal implementation of a screen projection method provided in an embodiment of this application.

[0208] Figure 10The illustration shows a scenario where a sending device (e.g., electronic device 100) receives a projected image frame, transmits the projected image frame to a receiving device (e.g., electronic device 200), and the receiving device displays the projected image frame from the sending device. The sending device can be an electronic device such as a mobile phone or tablet, and the receiving device can be an electronic device such as a tablet or computer.

[0209] The transmitting device may include an image synthesizer API, a virtual display module, and a multimedia encoder.

[0210] The virtual display module may include the VirtualDisplay module, which can provide the address of a specified storage area for the image synthesizer API and the multimedia encoder API.

[0211] The image compositing module API can call the SurfaceFlinger module. Once the SurfaceFlinger module is called, it can combine multiple application interfaces into a single projected image frame, compare the differences between the projected image frame and the previous projected image frame, and mark the changed application interface as a dirty area.

[0212] The multimedia encoder API can call the MediaCodec module, which encodes projected image frames and generates a data stream file. This data stream file includes dirty region range information (which allows the electronic device 200 to identify display data within the dirty region range). After encoding, the sending electronic device can send the generated transmission file to the receiving device via the RTSP protocol.

[0213] The receiving device may include a display management module. This module can decode the data stream file and parse the dirty region range. It then sends the application interface within the dirty region range to the GPU for rendering and display. For application interfaces outside the dirty region range, the receiving device will not send them to the GPU; instead, it will continue displaying the previous frame of the projected interface.

[0214] The following is a schematic diagram of the method flow provided in the embodiments of this application.

[0215] Figure 11 A flowchart illustrating the method of screen projection from electronic device 100 to electronic device 200 is shown.

[0216] S1101, electronic device 100 and electronic device 200 establish a first connection.

[0217] Once the electronic device 100 and the electronic device 200 establish the first connection, the electronic device 100 can transfer file data to the electronic device 200, or the electronic device 100 can project the application interface onto the electronic device 200 for display.

[0218] The first connection can be a wireless connection or a wired connection. When the first connection is a wireless connection, it can be any one of a Wi-Fi direct connection, a Wi-Fi LAN connection, or a Wi-Fi softAP connection.

[0219] S1102, the electronic device 100 can draw the first display data and the second display data in the projection image frame A (also known as the first projection image frame), and the electronic device 100 can also write the projection data A including the projection image frame A into the data stream file 1.

[0220] Electronic device 100 acquires display data from multiple projection windows and synthesizes projection image frame A. Electronic device 100 encodes projection image frame A using a specified encoding / decoding protocol (e.g., H.264) to obtain data stream file 1.

[0221] Specifically, electronic device 100 can acquire the first projection window (e.g., Figure 4H The first display data corresponding to the projection window 481 shown (e.g., Figure 8A The data shown is 801) and the second projection window (e.g., Figure 4H The second display data corresponding to the projection window 451 shown (for example, Figure 8A The data shown is 803). The electronic device 100 plots the first display data and the second display data on the projection image frame A (e.g., Figure 8A In the projected image frame A (shown as 800), the position information of the first display data corresponding to the first projected window, the position information of the second display data corresponding to the second projected window, and the dirty area range information A are recorded. Both the first and second display data are within the dirty area range.

[0222] Then, the electronic device 100 can encode the projected image frame A, etc., to obtain the data stream file 1.

[0223] For example, after the electronic device 100 obtains the projected image frame A, the electronic device 100 can compare... Figure 4G In the projected image frames 4 and A, electronic device 100 can obtain the dirty region range information A in projected image frame A. Specifically, electronic device 200 can determine from the dirty region range information A that display data 801, display data 802, and display data 803 are all within the dirty region range. Electronic device 100 can write projected image frame A, the position information of the projected page in projected image frame A, and the dirty region range information A into data stream file 1, and then send data stream file 1 to electronic device 200.

[0224] S1103, electronic device 100 sends data stream file 1 to electronic device 200.

[0225] The data stream file 1 may include the position information of the corresponding display data of the projection image frame A and multiple projection windows in the projection image frame A, as well as the dirty area range information A.

[0226] S1104, the electronic device 200 obtains the first display data and the second display data by cropping from the projected image frame A.

[0227] After receiving the data stream file 1, the electronic device 200 can decode the data stream file 1 to obtain the projection image frame A and the position information of the display data in the projection image frame A. Based on the position information of the display data corresponding to multiple projection windows in the projection image frame A and the dirty area range information A, the first display data and the second display data are cropped from the projection image frame A.

[0228] S1105, the electronic device 200 can render and display a first projection page in a first projection window based on the first display data, and render and display a second projection page in a second projection window based on the second display data.

[0229] The electronic device 200 renders the corresponding projection page using the GPU based on the cropped display data and stores the projection page corresponding to each projection window in the cache.

[0230] Specifically, the electronic device 200 can generate a first projection page based on the first display data (e.g., Figure 4H The projected page (as shown in page 492) is displayed in the first projected window. A second projected page is generated based on the second display data (e.g., ...). Figure 4H The projected page (474) is shown and displayed in the second projected window. At this time, the GPU utilization rate can be the first threshold.

[0231] For example, electronic device 200 receives a projection image frame A sent by electronic device 100. Figure 8A After receiving the projection data (shown in projection image frame 800), the electronic device 200 can crop, render, and display the image (display data 801, display data 802, display data 803) within the dirty area of ​​projection image frame A, as shown. Figure 4H As shown in the figure. During the rendering of the projection page, the electronic device 200 detected that the GPU utilization rate was at a first threshold (e.g., 80%).

[0232] S1106, the electronic device 100 can draw the third display data and the fourth display data in the projection image frame B (also known as the second projection image frame). The electronic device 100 can also obtain the position information of the display data in the projection image frame B, as well as the dirty area range information B of the projection image frame B. And write the projection data B, including the projection image frame B, the dirty area range information B, and the position information of the display data in the projection image frame B, into the data stream file 2.

[0233] Electronic device 100 acquires display data from multiple projection windows and synthesizes a projection image frame B. Electronic device 100 can encode the projection image frame B using a specified encoding / decoding protocol (e.g., H.264) to obtain a data stream file 2.

[0234] Specifically, electronic device 100 can obtain third display data corresponding to the first projection window (e.g., Figure 8B The data shown is 811), and the fourth display data of the second projection window (e.g., data 811). Figure 8B The data shown is 813). The electronic device 100 draws the third display data and the fourth display data in the projection image frame B, and records the position information of the third display data corresponding to the first projection window in the projection image frame B, and the position information of the fourth display data corresponding to the second projection window in the projection image frame B.

[0235] Simultaneously, the electronic device 100 can compare the projected image frame A and the projected image frame B to obtain dirty region range information B. If the electronic device 100 detects that the third display data is different from the first display data, it marks the third display data as being within the dirty region range. If the electronic device 100 detects that the fourth display data is the same as the second display data, it marks the fourth display data as not being within the dirty region range. After obtaining the dirty region range information B in the projected image frame B, the electronic device 100 can write the dirty region range information B into the data stream file 2.

[0236] For example, in Figures 4H-4I In the scenario shown, the electronic device 100 then obtains the projected image frame B. Figure 8B (See screen projection image frame 810). Electronic device 100 compares screen projection image frame A and screen projection image frame B to obtain the dirty region range information B of screen projection image frame B. That is, electronic device 100 finds that display data 811 and display data 812 are both within the dirty region range, while display data 813 is not within the dirty region range. Electronic device 100 can write screen projection image frame B, the position information of the projection page in screen projection image frame B, and the dirty region range information B into data stream file 2, and send data stream file 2 to electronic device 200.

[0237] S1107, Electronic device 100 sends data stream file 2 to electronic device 200.

[0238] The data stream file 2 includes a projection image frame B, the position information of the corresponding display data of multiple projection windows in the projection image frame B, and dirty area range information B.

[0239] S1108, the electronic device 200 detects that the third display data is within the dirty area range of the projection image frame B, and the fourth display data is not within the dirty area range of the projection image frame B. The electronic device 200 then obtains the third display data by cropping from the projection image frame B.

[0240] After obtaining the transport stream file 2, the electronic device 200 can decode the transport stream file 2 to obtain the projected image frame B and dirty area range information B, etc. Based on the position information of the display data corresponding to multiple projected windows in the projected image frame B and the dirty area range information B, the electronic device 200 can crop the third display data of the dirty area range.

[0241] S1109, the electronic device 200 can render and display a third projection page in a first projection window based on third display data, and display a cached second projection page in a second projection window.

[0242] Electronic device 200 can render third display data using a GPU to obtain a third projection page (e.g., Figure 4I The electronic device 200 displays the third screen-casting page (as shown in screen-casting page 493) in the first screen-casting window and stores it in the cache. Then, the electronic device 200 can display this third screen-casting page in the first screen-casting window. The electronic device 200 also displays the cached previous frame of the second screen-casting page (e.g., ...) in the second screen-casting window. Figure 4I The screen projection page shown is 474. During the rendering of the screen projection page, the electronic device 200 detects that the GPU usage rate is at the second threshold.

[0243] Wherein, assuming other conditions remain unchanged, the first threshold is greater than or equal to the second threshold. Thus, using the screen mirroring method provided in this application embodiment can effectively reduce GPU usage.

[0244] For example, electronic device 200 can crop the display data 811 from the projection image frame B, render it to obtain projection page 493 and display it in projection window 491, crop the display data 812, render it to obtain projection page 486 and display it in projection window 481, and continue to display projection page 474 in projection window 451, such as... Figure 4IAs shown, during the rendering of the projection page, the electronic device 200 detects that the GPU utilization rate is at a first threshold (e.g., 50%). Since only display data 811 and display data 812 are in the dirty region in the projection image frame B, the electronic device 200 only needs to render display data 811 and display data 812, effectively reducing the GPU utilization rate.

[0245] In one possible implementation, electronic device 100 can obtain first display data corresponding to the first projection window on electronic device 200 and second display data corresponding to the second projection window on electronic device 200. Electronic device 100 can send the first display data and the second display data to electronic device 200. Electronic device 200 can render and display a first projection page in the first projection window based on the first display data, and render and display a second projection page in the second projection window based on the second display data.

[0246] Subsequently, electronic device 100 can acquire the third display data corresponding to the first projection window on electronic device 200 and the fourth display data corresponding to the second projection window on electronic device 200. Electronic device 100 detects that the third display data is different from the first display data, while the fourth display data is the same as the second display data. Electronic device 100 can compare the third display data with the first display data to obtain the region C where the third display data differs from the first display data, and record the position information of region C. Electronic device 100 can then send the third display data, the fourth display data, and the position information of region A to electronic device 200.

[0247] Electronic device 200 can extract display data from region C based on the position information of region C from the third display data. Electronic device 200 can display a cached first projection page in a first projection window and render the obtained display data from region C at the position of region C in the first projection page. A cached second projection page is then displayed in a second projection window. In this way, electronic device 200 can more accurately obtain the regions that change between the two projection image frames, further reducing GPU usage.

[0248] Optionally, the electronic device 100 may send only the third display data and the location information of area A.

[0249] Optionally, electronic device 100 may send only the display data in region A.

[0250] For example, such as Figures 4H-4IAs shown, electronic device 100 obtains a projection image frame 800. Electronic device 100 sends the display data of the projection image frame 800 to electronic device 200. After receiving the display data of the projection image frame 800, electronic device 200 can cut and render the display pages corresponding to each projection window in the projection image frame 800. Then, electronic device 100 obtains a projection image frame 810. Electronic device 100 compares projection image frame 810 with projection image frame 800 and finds that the entire image of display page 811 has changed, marking the entire page area as a dirty area. It also finds that part of the image of display page 812 has changed, marking the playback window area of ​​that page as a dirty area. Electronic device 100 can record the position information of the dirty area. This position information can include the coordinates of the upper left corner of the dirty area, the horizontal distance of the dirty area, and the vertical distance of the dirty area. For example, the location information of the dirty area range of the projected image frame 810 can be (15, 180, 55, 160) or (80, 170, 55, 50). Here, (15, 180, 55, 160) can represent the entire dirty area range of the display page 811, extending 55 pixels to the left from coordinate point (15, 180) and downwards for 160 pixels. Similarly, (80, 170, 55, 50) can represent the entire dirty area range of the display page 812, extending 55 pixels to the left from coordinate point (80, 170) and downwards for 50 pixels. Electronic device 100 can send both the projected image frame 810 and the location information of the dirty area range to electronic device 200. Electronic device 200 can extract, render, and stitch the image data within the dirty area range to the corresponding position in the previous projected image frame based on the received location information. The stitched image is then sent to the corresponding projection window for display.

[0251] In one possible implementation, electronic device 100 can obtain first display data corresponding to the first projection window on electronic device 200 and second display data corresponding to the second projection window on electronic device 200. Electronic device 100 can send the first and second display data to electronic device 200. Electronic device 200 can render and display a first projection page in the first projection window based on the first display data, and render and display a second projection page in the second projection window based on the second display data. Then, electronic device 100 can obtain third display data corresponding to the first projection window on electronic device 200 and fourth display data corresponding to the second projection window on electronic device 200. Electronic device 100 detects that the third display data is different from the first display data, and the fourth display data is the same as the second display data. Electronic device 100 can send the third display data to electronic device 200. Electronic device 200 can render and display a third projection page in the first projection window based on the third display data, and display the cached second projection page in the second projection window.

[0252] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A screen projection system, characterized in that, include: First electronic device and second electronic device; wherein... The first electronic device is used to acquire first display data corresponding to the first projection window on the second electronic device and second display data corresponding to the second projection window on the second electronic device; The first electronic device is further configured to draw the first display data and the second display data in the first projection image frame, and record the area range of the first display data in the first projection image frame and the area range of the second display data in the first projection image frame; The first electronic device is further configured to send the first projection image frame, the area range of the first display data in the first projection image frame, and the area range of the second display data in the first projection image frame to the second electronic device; The second electronic device is configured to crop out the first display data from the first projection image frame based on the area range of the first display data in the first projection image frame, and to crop out the second display data from the first projection image frame based on the area range of the second display data in the first projection image frame; The second electronic device is further configured to render and display a first projection page in the first projection window based on the first display data, and to render and display a second projection page in the second projection window based on the second display data; The first electronic device is further configured to acquire third display data corresponding to the first projection window on the second electronic device and fourth display data corresponding to the second projection window on the second electronic device; The first electronic device is further configured to draw the third display data and the fourth display data in the second projection image frame, record the area range of the third display data in the second projection image frame and the area range of the fourth display data in the second projection image frame, and determine the range of dirty areas in the second projection image frame where the image content differs from that in the first projection image frame. The first electronic device is further configured to send the first projection image frame, the area range of the first display data in the first projection image frame, the area range of the second display data in the first projection image frame, and the dirty area range to the second electronic device; The second electronic device is further configured to crop out the third display data from the second projection image frame based on the area range of the third display data in the second projection image frame, and to crop out the fourth display data from the second projection image frame based on the area range of the fourth display data in the second projection image frame; The second electronic device is further configured to, when it is determined that the third display data is different from the first display data and the fourth display data is the same as the second display data, render and display a third projection page in the first projection window based on the third display data, and display the cached second projection page in the second projection window; The determination that the third display data is different from the first display data and the fourth display data is the same as the second display data specifically includes: When the dirty area range overlaps with the area range of the third display data in the second projection image frame, but does not overlap with the area range of the fourth display data in the second projection image frame, it is determined that the third display data is different from the first display data, and the fourth display data is the same as the second display data.

2. The system according to claim 1, characterized in that, The second electronic device is further configured to, when it is determined that the third display data is different from the first display data and the fourth display data is different from the second display data, render and display a third projection page in the first projection window based on the third display data, and render and display a fourth projection page in the second projection window based on the fourth display data.

3. The system according to claim 1, characterized in that, The second electronic device is further configured to, when it is determined that the third display data is the same as the first display data and the fourth display data is the same as the second display data, display the cached first projection page in the first projection window and display the cached second projection page in the second projection window.

4. The system according to claim 1, characterized in that, The first electronic device is further configured to establish a wireless fidelity Wi-Fi direct connection with the second electronic device before acquiring the first display data corresponding to the first projection window on the second electronic device and the second display data corresponding to the second projection window on the second electronic device; The first electronic device is specifically used to send the first display data and the second display data to the second electronic device via the Wi-Fi Direct connection; The first electronic device is specifically used to send the first projection image frame, the area range of the first display data in the first projection image frame, the area range of the second display data in the first projection image frame, and the dirty area range to the second electronic device via the Wi-Fi direct connection.

5. A screen projection method, applied to a second electronic device, characterized in that, include: The second electronic device receives a first projection image frame, a region of first display data in the first projection image frame, and a region of second display data in the first projection image frame from the first electronic device; wherein, the first projection image frame includes the first display data and the second display data; The second electronic device crops out the first display data from the first projection image frame based on the area range of the first display data in the first projection image frame, and crops out the second display data from the first projection image frame based on the area range of the second display data in the first projection image frame; The second electronic device renders and displays a first projection page in a first projection window based on the first display data, and renders and displays a second projection page in a second projection window based on the second display data; The second electronic device receives a second projection image frame, a region of third display data in the second projection image frame, a region of fourth display data in the second projection image frame, and a dirty region in the second projection image frame whose image content differs from that in the first projection image frame, sent by the first electronic device; wherein, the second projection image frame includes the third display data and the fourth display data; The second electronic device crops out the third display data from the second projection image frame based on the area range of the third display data in the second projection image frame, and crops out the fourth display data from the second projection image frame based on the area range of the fourth display data in the second projection image frame; When the second electronic device determines that the third display data is different from the first display data and the fourth display data is the same as the second display data, the second electronic device renders and displays the third projection page in the first projection window based on the third display data, and displays the cached second projection page in the second projection window. The second electronic device determines that the third display data is different from the first display data, and that the fourth display data is the same as the second display data, specifically including: When the dirty area range overlaps with the area range of the third display data in the second projection image frame, but does not overlap with the area range of the fourth display data in the second projection image frame, the second electronic device determines that the third display data is different from the first display data, and the fourth display data is the same as the second display data.

6. The method according to claim 5, characterized in that, The method further includes: When the second electronic device determines that the third display data is different from the first display data and the fourth display data is different from the second display data, the second electronic device renders and displays the third projection page in the first projection window based on the third display data, and renders and displays the fourth projection page in the second projection window based on the third display data.

7. The method according to claim 5, characterized in that, The method further includes: When the second electronic device determines that the third display data is the same as the first display data and the fourth display data is the same as the second display data, it displays the cached first screen projection page in the first screen projection window and the cached second screen projection page in the second screen projection window.

8. The method according to claim 5, characterized in that, Before the second electronic device receives the first projection image frame, the area range of the first display data in the first projection image frame, and the area range of the second display data in the first projection image frame sent by the first electronic device, the method further includes: The second electronic device establishes a wireless fidelity Wi-Fi direct connection with the first electronic device; The second electronic device receives a first projection image frame, the area range of the first display data in the first projection image frame, and the area range of the second display data in the first projection image frame sent by the first electronic device, specifically including: The second electronic device receives, via the Wi-Fi direct connection, a first projection image frame, the area range of the first display data in the first projection image frame, and the area range of the second display data in the first projection image frame sent by the first electronic device; The second electronic device receives a second projection image frame, a region of third display data in the second projection image frame, a region of fourth display data in the second projection image frame, and a dirty region in the second projection image frame where the image content differs from that in the first projection image frame, specifically including: The second electronic device receives, via the Wi-Fi Direct Connection, a second projection image frame sent by the first electronic device, the area range of the third display data in the second projection image frame, the area range of the fourth display data in the second projection image frame, and the dirty area range in the second projection image frame where the image content differs from that in the first projection image frame.

9. An electronic device, a second electronic device, characterized in that, include: One or more processors, a Wi-Fi module, and one or more memories; wherein the one or more memories and the Wi-Fi module are coupled to the one or more processors, the one or more memories being used to store computer program code, the computer program code including computer instructions, which, when the one or more processors execute the computer instructions, cause the second electronic device to perform the method as described in any one of claims 5-8.

10. A computer storage medium, characterized in that, Includes computer instructions that, when executed on a second electronic device, cause the second electronic device to perform the method as described in any one of claims 5-8.

11. A computer program product, characterized in that, When the computer program product is run on the second electronic device, the second electronic device performs the method as described in any one of claims 5-8.

Images