Method for smoothly displaying a picture in screen projection, related device and system

Abstract

The application discloses a method for smoothly displaying a picture in screen projection, a related device and a system. In the method, in a mirror screen projection process, the screen projection device and the projected device can perform image prediction according to the latest received image frame, and display the predicted image frame when frame loss occurs. According to the application, the projected device side can continuously display the picture, maintain the continuity of the picture in vision, ensure the high frame rate of the projected device side, and avoid the problem of lag. In addition, the image prediction according to the latest received image frame can guarantee the smoothness and stability of the picture, avoid the visual jump, and provide a good screen projection experience for the user.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to methods, related devices and systems for smoothly displaying images during screen projection. Background Technology

[0002] Screen mirroring is a widely used function on electronic devices, including mirroring and online mirroring. In mirroring, the mirroring device captures the content displayed on its own screen and sends it to the recipient device, ensuring the recipient device displays the same content. During mirroring, the smoothness of the content display on the recipient device is a major factor affecting the user's visual experience. Therefore, ensuring smooth display of the mirrored content is an important research direction for improving user experience. Summary of the Invention

[0003] This application provides a method, related apparatus and system for smoothly displaying images during screen mirroring, supporting the display of smooth and continuous images on the screen-mirrored device, avoiding any stuttering or jumping sensations.

[0004] In a first aspect, a method for smoothly displaying images during screen projection is provided, applied to a communication system including a first device and a second device. The method includes: the first device and the second device establishing a communication connection; the first device capturing the content displayed on the screen to obtain a first image frame, and sending the first image frame to the second device; the second device receiving the first image frame; the second device sequentially displaying: the first image frame and a first predicted image frame; the first predicted image frame being obtained by the second device based on the first image frame.

[0005] By implementing the method provided in the first aspect, during screen mirroring, the second device, i.e., the device being mirrored to, displays both the received and predicted image frames. This allows for continuous image display, maintaining visual continuity and avoiding stuttering. Furthermore, image prediction based on the latest received image frame ensures smoothness and stability, preventing visual jumps. From the user's perspective, this results in a smooth and continuous display without any stuttering or jumps, providing a superior screen mirroring experience.

[0006] In conjunction with the first aspect, in some implementations, the communication connection established between the first device and the second device can be a Wi-Fi connection, Bluetooth connection, NFC connection, or remote connection, etc. The connection between the first device and the second device can be established based on protocols such as Miracast, the Digital Living Network Alliance (DLNA) protocol, and AirPlay.

[0007] In conjunction with the first aspect and any of the above embodiments, in some embodiments, before obtaining the first image frame, the first device may capture the content displayed on the display screen to obtain a second image frame and send the second image frame to the second device; the second device receives the second image frame; the second device also displays the second image frame before displaying the first image frame; wherein, the first predicted image frame is obtained by the second device based on the first image frame and the second image frame.

[0008] Through the above implementation method, the second device, i.e. the device being projected, can obtain the predicted image frame based on the two most recently received image frames.

[0009] In conjunction with the previous embodiment, the time point at which the second device receives the second image frame and the time point at which the first device receives the second image frame are within a first time duration. That is to say, only image frames with lower latency received by the second device can be displayed. This ensures low latency for screen mirroring, giving users the feeling that the second and first devices are displaying images synchronously, thus providing a better screen mirroring experience.

[0010] In conjunction with the first aspect and any of the above embodiments, in some embodiments, after the first device obtains the first image frame, it can again capture the content displayed on the screen to obtain a third image frame, and send the third image frame to the second device; the second device does not receive the third image frame, or the second device does not receive the third image frame within a first time period after the first device captures the third image frame. Through this embodiment, the second device can display a predicted image frame when frame loss occurs, such as not receiving an image frame or not receiving an image frame within a specified time. This allows the second device to continuously display the image, maintaining visual continuity and avoiding stuttering issues.

[0011] In conjunction with the first aspect and any of the above embodiments, in some embodiments, the second device may obtain the first predicted image frame based on the first image frame in any of the following cases:

[0012] Case 1: After the first device and the second device establish a communication connection, the second device obtains the first predicted image frame based on the first image frame.

[0013] Case 2: When the communication quality corresponding to the communication connection is lower than the threshold, the second device obtains the first predicted image frame based on the first image frame.

[0014] If the communication quality between the first and second devices is below a certain value, it indicates poor communication quality between them. This poor communication quality could lead to the loss of image frames sent from the first device to the second device during communication. In other words, scenario 2 is highly likely to result in frame loss. Therefore, obtaining the first predicted image frame in scenario 2 can effectively avoid the negative user experience caused by potential frame loss.

[0015] Scenario 3: The first device runs the first application in the foreground and scrolls the content on the screen at a speed greater than the first value.

[0016] In scenario 3, the content on the first device's display changes rapidly, and the first device will capture the screen content at a high projection frame rate. If frame drops occur at a high projection frame rate, it will cause very noticeable visual stuttering for the user. Therefore, capturing the first predicted image frame in scenario 3 can effectively avoid the poor user experience caused by frame drops at high projection frame rates.

[0017] In some implementations, the user operation received by the first device can trigger the above-described situation 3. Specifically, the method of the first aspect may further include: before the second device displays the first predicted image frame, the first device runs a first application and receives a first operation; the first device responds to the first operation by scrolling the content on the display screen at a speed greater than a first value; the first device sends the application information of the first application and the operation information of the first operation to the second device; the second device receives the application information of the first application and the operation information of the first operation, and obtains the first predicted image frame based on the first image frame.

[0018] In some implementations, the user operation received by the second device can trigger situation 3 described above. Specifically, the method of the first aspect may further include: before the second device displays the first predicted image frame, the first device runs a first application and sends the application information of the first application to the second device; the second device receives a second operation and sends the operation information of the second operation to the first device, triggering the first device to scroll the content on the display screen at a speed greater than a first value; the second device obtains the first predicted image frame based on the first image frame.

[0019] The first type of application is one that can respond to user actions by swiping or scrolling to display content in the user interface. Examples include browsers, social networking applications, reading applications, and so on.

[0020] Case 4: If the difference between the display frame rate of the image frame sent by the first device and the projection frame rate of the content displayed on the screen captured by the first device is greater than the second value, the second device obtains the first predicted image frame based on the first image frame.

[0021] The closer the display frame rate of the second device is to the projection frame rate of the first device, the better the mirrored projection effect will be for the user. The greater the difference between the display frame rate of the second device and the projection frame rate of the first device, the more severe the frame drop problem will be during the mirrored projection process. In scenario 4, predicted image frames can be obtained when a relatively serious frame drop problem begins to occur, avoiding the poor user experience caused by subsequent continuous frame drop.

[0022] In conjunction with the first aspect and any of the above embodiments, in some embodiments, the time point at which the second device receives the first image frame and the time point at which the first device receives the first image frame are within a first time duration. That is to say, only image frames with low latency received by the second device can be displayed, which can ensure low latency in screen mirroring and give users the feeling that the second device and the first device are displaying images synchronously, providing users with a better screen mirroring experience.

[0023] In conjunction with the first aspect and any of the above embodiments, in some embodiments, the first predicted image frame is: based on the first image frame, the content displayed in the motion region is moved in the motion region according to the motion vector, and then the predicted data is used to fill the empty region to obtain the image.

[0024] In this context, the motion region refers to the area in the first image frame and the fourth image frame where different content is displayed; the movement vector is the vector that moves the target content from its position in the fourth image frame to its position in the first image frame; and the idle region is the area in the motion region where no content is displayed after the content displayed in the motion region has been moved. If the second device obtains the first predicted image frame based on the first image frame, then the fourth image frame is the image frame obtained by the first device from the most recent capture of the content displayed on the screen before the first image frame; if the second device obtains the first predicted image frame based on the first image frame and the second image frame, then the fourth image frame is the second image frame most recently displayed by the second device before displaying the first image frame. The prediction data is obtained by the second device based on the content displayed in the idle region of the first image frame.

[0025] If the second device obtains the first predicted image frame based on the first image frame, the second device can determine the aforementioned motion area and movement vector based on the first image frame, the application information running on the first device, and the operation information that triggers the sliding display interface of the first device.

[0026] If the second device obtains the first predicted image frame based on the first image frame and the second image frame, the second device can determine the aforementioned motion region and motion vector by comparing the first image frame and the second image frame.

[0027] In the above implementation method, the second device performs image prediction based on the latest received image frame, which ensures the smoothness and stability of the projected image and avoids visual jumps. This allows users to see a smooth image without any jerking.

[0028] In conjunction with the previous embodiment, in some embodiments, the second device can obtain prediction data based on the content displayed in the idle area of ​​the first image frame through any one or more of the following methods:

[0029] 1. The second device performs blurring processing on the content displayed in the idle area of ​​the first image frame. The blurring processing may include image processing methods such as mean blur, median blur, Gaussian blur, bilateral blur, surface blur, box blur, double blur, bokeh blur, tilt-shift blur, aperture blur, grain blur, radial blur, and directional blur.

[0030] 2. The second device directly uses the content displayed in the idle area of ​​the first image frame as prediction data.

[0031] 3. The second device uses a neural network algorithm to predict the content displayed in the idle area of ​​the first image frame and obtain prediction data.

[0032] 4. The second device obtains prediction data from previously cached image frames.

[0033] Through the above implementation method, the second device can obtain prediction data based on the content displayed by the first image frame in the idle area, thereby obtaining the first predicted image frame. This method of obtaining the first predicted image frame based on the first image frame can ensure the smoothness and stability of the projected screen, avoid visual jumps, and allow users to see a smooth screen without any jerking sensation.

[0034] In conjunction with the first aspect and any of the above embodiments, in some embodiments, the display screen of the second device includes a projection area for sequentially displaying a first image frame and a first predicted image frame. The projection area occupies part or all of the display screen of the second device. When the projection area occupies the entire display screen, the user can obtain an immersive projection experience; when the projection area occupies part of the display screen, the second device can utilize other areas of the display screen to display the user interface provided by the second device itself, without affecting the user's operation of the second device.

[0035] In some implementations, when the projection area occupies part of the display screen, the second device can also respond to user operations by adjusting the position, size, shape, etc. of the projection area.

[0036] In conjunction with the first aspect and any of the above embodiments, in some embodiments, after the second device displays the first predicted image frame, it displays a second predicted image frame, which is obtained by the second device based on the first predicted image frame. This is equivalent to the second device continuously predicting and displaying multiple predicted image frames, ensuring continuous display of the image for a period of time, maintaining visual continuity, and avoiding stuttering issues. This ensures the smoothness and stability of the image on the second device for a period of time, avoiding visual jumps.

[0037] In conjunction with the first aspect and any of the above embodiments, in some embodiments, the faster the content displayed on the screen of the first device changes, the higher the frame rate at which the first device captures the content displayed on the screen. This allows the first device to sensitively capture the changing process of the image on the screen, thereby enabling the second device to also present a corresponding response process, avoiding abrupt changes in the second device.

[0038] In conjunction with the first aspect, in some embodiments, the display frame rate of the second device sequentially displaying the first image frame and the first predicted image frame is equal to the projection frame rate of the content displayed on the screen captured by the first device. When the display frame rate and the projection frame rate are consistent, the user can see the optimal mirror projection effect.

[0039] Secondly, a method for smoothly displaying images during screen projection is provided, applied to a second device. The method includes: the second device establishing a communication connection with a first device; the second device receiving a first image frame sent by the first device, the first image frame being obtained by the first device capturing content displayed on the screen; the second device sequentially displaying: the first image frame and a first predicted image frame; the first predicted image frame being obtained by the second device based on the first image frame.

[0040] The method of the second aspect can refer to the operation performed by the second device in the first aspect or any embodiment of the first aspect. The technical effects that the method of the second aspect can achieve can also refer to the technical effects of the operation performed by the second device in the first aspect or any embodiment of the first aspect.

[0041] Thirdly, an electronic device is provided, comprising: a memory and one or more processors; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, and the one or more processors invoking the computer instructions to cause the electronic device to perform a method as described in the second aspect or any embodiment of the second aspect.

[0042] Fourthly, embodiments of this application provide a communication system, including a first device and a second device, wherein the second device is used to perform the method as described in the second aspect or any of the embodiments of the second aspect.

[0043] Fifthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform a method as described in the second aspect or any embodiment of the second aspect.

[0044] In a sixth aspect, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform the method of the second aspect or any of the embodiments of the second aspect.

[0045] By implementing the technical solution provided in this application, during screen mirroring, the projected device can predict the image based on the latest received image frame and display the predicted image frame when frame loss occurs. Implementing this solution allows the projected device to continuously display the image, maintaining visual continuity, ensuring a high frame rate, and avoiding stuttering. Furthermore, image prediction based on the latest received image frame ensures smoothness and stability of the image, preventing visual jumps and providing users with a superior screen mirroring experience. Attached Figure Description

[0046] Figure 1 The architecture of the communication system provided in the embodiments of this application;

[0047] Figure 2A A hardware structure diagram of the electronic device provided in the embodiments of this application;

[0048] Figure 2B A software structure diagram of the electronic device provided in the embodiments of this application;

[0049] Figure 3 A flowchart illustrating a method for smoothly displaying a screen during projection, as provided in this application embodiment;

[0050] Figures 4A-4D The user interface involved when starting the screen mirroring function on the source device 100;

[0051] Figure 4E The user interface involved when enabling the screen mirroring function on the terminal device 200;

[0052] Figures 5A-5E This refers to the user interface used by the source device 100 during the screen mirroring process.

[0053] Figure 6 This is a schematic diagram of the display queue and the prediction image queue provided in the embodiments of this application;

[0054] Figure 7 A schematic diagram of the predicted image provided in an embodiment of this application;

[0055] Figures 8A-8D This refers to the user interface involved in the screen mirroring process of the end device 200. Detailed Implementation

[0056] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0057] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0058] The term "user interface (UI)" used in the following embodiments of this application refers to the medium interface through which an application or operating system interacts and exchanges information with the user. It realizes the conversion between the internal form of information and the form that the user can accept. The user interface is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on the electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets displayed on the screen of an electronic device.

[0059] During screen mirroring, scenarios with high frame rates may occur. For example, during mirroring, when the content displayed on the screen of the mirroring device changes rapidly, the frame rate of the screen content captured by the mirroring device is high. In this situation, if there are issues such as network latency or low device encoding / decoding speed, frame drops may occur during the process of the mirroring device sending the screen content to the recipient device. This results in a lower display frame rate on the recipient device, causing the display to be choppy and exhibiting noticeable visual stuttering.

[0060] To address the aforementioned issues, the following embodiments of this application provide a method, related apparatus, and system for smooth display of images during screen mirroring. In this method, during screen mirroring, if the screen mirroring frame rate on the screen mirroring device side is higher than a threshold, the screen mirroring device side can perform image prediction based on the latest received image frame, and display the predicted image frame when frame loss occurs.

[0061] By implementing the above method, the device being projected displays both the received and predicted image frames, ensuring continuous image display, visual continuity, and a high frame rate, thus avoiding stuttering. Furthermore, image prediction based on the latest received image frame guarantees smoothness and stability, preventing visual jumps. From the user's perspective, this results in a smooth and continuous display without stuttering or jumps, providing a superior projection experience.

[0062] In the methods provided in the following embodiments of this application, screen mirroring refers to the process by which a screen mirroring device captures the content displayed on its own screen and sends it to a target device, so that the target device displays the same content as the screen mirroring device. The technologies used for screen mirroring may include, but are not limited to, wireless fidelity (Wi-Fi), Bluetooth, near field communication (NFC), mobile communication technologies, or wired technologies, etc. The protocols used for screen mirroring may include, but are not limited to, Miracast, the Digital Living Network Alliance (DLNA) protocol, AirPlay, etc.

[0063] Mirroring is merely a term used in the embodiments of this application, and its meaning has been described in these embodiments. Its name does not constitute any limitation on these embodiments. For example, mirroring can also be referred to by other terms, such as collaborative casting, mirror casting, wireless casting, etc.

[0064] In the process of screen mirroring, the screen mirroring device can also be referred to as the source-side device, and the screen being mirrored can also be referred to as the endpoint-side device. In the following embodiments, the source-side device and the endpoint-side device will be used as examples for description. In the embodiments of this application, the source-side device can also be referred to as the first device, and the endpoint-side device can also be referred to as the second device.

[0065] In the methods provided in the following embodiments of this application, the screen casting frame rate refers to the number of frames of screen content captured by the source device per unit time. The unit of screen casting frame rate can be frames per second (FPS) or hertz (Hz). The size of the screen casting frame rate is related to the speed of change of the image on the display screen of the source device; the faster the image changes, the higher the screen casting frame rate at which the source device captures the screen content.

[0066] Scenarios where the source device can achieve a high frame rate during screen mirroring include those where the content displayed on the source device changes rapidly due to the user quickly swiping across the screen. For example, when the source device is running applications such as browsers or social media apps, if the user quickly swipes across the user interface on the screen, the image on the screen will change rapidly, resulting in a high frame rate for the source device during screen mirroring.

[0067] In the following embodiments of this application, frame loss refers to the loss of the screen-projected content (i.e., image frames) captured by the source device during transmission to the end device, or the active discarding by the end device. Causes of frame loss may include, but are not limited to: poor connection conditions between the source and end devices, such as poor network quality (e.g., low network speed), low wired bandwidth, low image encoding efficiency on the source side, and low image decoding efficiency on the end device.

[0068] Below, we will first introduce the communication system 10 for screen projection provided in the embodiments of this application.

[0069] Figure 1 An example of the architecture of communication system 10 is shown.

[0070] like Figure 1 As shown, the communication system 10 includes: source-side device 100 and end-side device 200.

[0071] In this embodiment, the source-side device 100 can establish a communication connection with the end-side device 200. This communication connection can be a Wi-Fi connection, Bluetooth connection, NFC connection, or remote connection, or it can be a wired connection such as a data cable-based connection; this embodiment does not impose any limitations on this.

[0072] The source-side device 100 may include, but is not limited to, smartphones, tablets, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices (such as smartwatches and smart glasses), etc. Exemplary embodiments of the electronic device include, but are not limited to, devices equipped with… Portable electronic devices operating systems such as Linux or others. These electronic devices can also be other portable electronic devices, such as laptops. It should also be understood that in some other embodiments, the electronic device may not be a portable electronic device, but rather a desktop computer.

[0073] The source-side device 100 includes a display screen that can display content locally on the source-side device 100 or content from the network. The display screen can also receive various types of gestures input by the user, such as swipe gestures, click gestures, drag gestures, pinch gestures, etc. The source-side device 100 can respond to these user-input gestures by changing the content displayed on the screen.

[0074] The edge device 200 can be a tablet computer, television, smart screen, in-vehicle device, or electronic billboard, etc. The edge device 200 may have a larger display screen compared to the source device 100. In some embodiments, when the edge device 200 is a television, it can be used in conjunction with a TV box, which converts the received digital signal into an analog signal and sends it to the television for display. In some embodiments, the edge device 200 can be a television with built-in digital-to-analog conversion capabilities, or a television equipped with a TV box. In some embodiments, when the edge device 200 is a television or smart screen, it can also be used in conjunction with a remote control. The remote control and the edge device 200 can communicate via infrared signals.

[0075] In this embodiment, after the source device 100 and the terminal device 200 establish a communication connection, they can perform a screen mirroring process. The source device 100 can display corresponding content on the display screen according to the user's input operation, and determine the screen mirroring frame rate according to the change speed of the content on the display screen. The source device 100 uses the screen mirroring frame rate to capture the content displayed on the display screen and sends the content to the terminal device 200 through the communication connection.

[0076] During the aforementioned screen mirroring process, the source device 100 can notify the end device 200 of its determined screen mirroring frame rate based on the communication connection with the end device 200. The source device 100 can also send its own operating status, such as information about the running applications, to the end device 200 based on the communication connection with the end device 200.

[0077] The edge device 200 can perform image prediction based on the latest received image frame to obtain the predicted image frame. The timing or scenario for the edge device 200 to predict the image frame can be found in the detailed description of the subsequent method embodiments, and will not be repeated here.

[0078] Figure 2A This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. This electronic device can be... Figure 1 The source-side device 100 in the communication system shown can also be the end-side device 200.

[0079] like Figure 2A As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0080] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0081] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0082] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0083] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0084] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0085] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0086] The mobile communication module 150 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G in electronic devices. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0087] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0088] The wireless communication module 160 can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, frequency modulate and amplify them, and then convert them into electromagnetic waves for radiation via antenna 2.

[0089] In some embodiments, antenna 1 of the electronic device is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling the electronic device to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0090] Electronic devices implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0091] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be manufactured using organic light-emitting diodes (OLEDs), active-matrix organic light-emitting diodes (AMOLEDs), flexible light-emitting diodes (FLEDs), miniled, microled, micro-OLEDs, quantum dot light-emitting diodes (QLEDs), etc. In some embodiments, the electronic device may include one or N displays 194, where N is a positive integer greater than 1.

[0092] Electronic devices can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0093] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device is selecting a frequency, a DSP can perform a Fourier transform on the frequency energy.

[0094] Video codecs are used to compress or decompress digital video. Electronic devices can support one or more video codecs. This allows the electronic device to play or record video in various encoded formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0095] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0096] Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), and double data rate synchronous dynamic random access memory (DDR SDRAM, such as fifth-generation DDR SDRAM, which is generally called DDR5 SDRAM). Non-volatile memory can include disk storage devices and flash memory.

[0097] Flash memory can be classified according to its operating principle, including NOR FLASH, NAND FLASH, 3D NAND FLASH, etc.; according to the level of the storage cell, including single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc.; and according to the storage specification, including universal flash storage (UFS) and embedded multimedia card (eMMC), etc.

[0098] The random access memory can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0099] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110.

[0100] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0101] Electronic devices can implement audio functions such as music playback and recording through audio modules 170, speakers 170A, receivers 170B, microphones 170C, headphone jacks 170D, and application processors.

[0102] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. The electronic device determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, the electronic device detects the intensity of the touch operation based on pressure sensor 180A. The electronic device can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0103] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of the electronic device, in a different position than display screen 194.

[0104] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. The electronic device can receive button input and generate key signal inputs related to user settings and function control of the electronic device.

[0105] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0106] When the source-side device 100 is implemented as Figure 2A When the electronic device shown is used:

[0107] The display screen 194 is used to display content from the source device 100 locally, or content from the network. The display screen 194 can also receive various gesture operations input by the user and display different content in response to the gesture operation.

[0108] The wireless communication module 160 is used to establish a communication connection with the end-side device 200. This communication connection can be a wireless communication connection such as a Wi-Fi connection or a Bluetooth connection.

[0109] The processor 110 is used to determine the projection frame rate based on the speed of change of the content displayed on the display screen 194, and to capture the content displayed on the display screen 194 and obtain image frames using the projection frame rate.

[0110] The wireless communication module 160 is used to send the projection frame rate and image frames determined by the processor 110 to the end-side device 200 based on the communication connection between the module and the end-side device 200. In some embodiments, the wireless communication module 160 can also be used to send information about the application running on the source-side device 100 to the end-side device 200.

[0111] When the end-side device 200 is implemented as Figure 2A When the electronic device shown is used:

[0112] The wireless communication module 160 is used to establish a communication connection with the source-side device 100. This communication connection can be a wireless communication connection such as a Wi-Fi connection or a Bluetooth connection.

[0113] The wireless communication module 160 is also used to receive screen projection frame rate, captured image frames, etc., sent by the source-side device 100 based on the communication connection between the source-side device 100 and the source-side device 100. In some embodiments, the wireless communication module 160 can also receive information about the application running on the source-side device 100 sent by the source-side device 100.

[0114] The processor 110 is used to perform image prediction based on the latest received image frame to obtain the predicted image frame.

[0115] The processor 110 is also used to determine whether there is frame loss based on the image frames sent by the source device 100 and the projection frame rate. If frame loss occurs, the predicted image frames are displayed.

[0116] Figure 2A The software system of the electronic device in the embodiments shown in this application can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of the electronic device.

[0117] Figure 2B This is a software structure block diagram of an electronic device according to an embodiment of this application.

[0118] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0119] The application layer can include a series of application packages.

[0120] like Figure 2B As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0121] In this embodiment, the application package may include a screen mirroring application. The screen mirroring application in the source device 100 is used to support the smooth display of the screen in subsequent screen mirroring operations performed by the source device 100 in the method embodiment, and the screen mirroring application in the terminal device 200 is used to support the smooth display of the screen in subsequent screen mirroring operations performed by the terminal device 200 in the method embodiment.

[0122] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0123] like Figure 2B As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0124] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0125] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0126] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0127] A phone manager is used to provide communication functions for electronic devices. For example, it manages call status (including connection and disconnection).

[0128] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0129] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0130] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0131] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0132] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0133] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0134] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0135] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0136] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0137] A 2D graphics engine is a graphics engine for 2D drawing.

[0138] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0139] The following is based on Figure 1 The communication system 10 shown, Figure 2A and Figure 2B The electronic device shown illustrates a method for smoothly displaying images during screen projection, as provided in the embodiments of this application.

[0140] Figure 3 The flowchart of the method for smoothly displaying the screen during projection is illustrated by example.

[0141] like Figure 3 As shown, the method may include the following steps:

[0142] S101, the source-side device 100 and the end-side device 200 establish a communication connection.

[0143] In this embodiment of the application, the source device 100 can detect user input operations and, in response to the user operations, enable one or more of WLAN, Bluetooth, NFC or mobile networks in the wireless communication module 160, and can discover other devices that can establish communication connections for screen mirroring through one or more of the wireless communication technologies of Wi-Fi, Bluetooth, NFC and mobile networks.

[0144] In some embodiments, after discovering multiple devices that can establish communication connections for screen mirroring, the source-side device 100 can display the identifiers of these multiple devices in the user interface, allowing the user to select one or more devices to establish a communication connection.

[0145] Figure 4A and Figure 4B The process of establishing a communication connection between source-side device 100 and end-side device 200 is illustrated by way of example.

[0146] Figure 4A An exemplary user interface 41 for displaying installed applications is shown on the source-side device 100.

[0147] User interface 41 displays: a status bar, a calendar and time indicator, a weather indicator, a page indicator, a tray with icons of commonly used applications, other application icons, etc. However, this is not the only feature; in some embodiments, Figure 4A The user interface 41 shown may also include a navigation bar, a sidebar, etc. In some embodiments, Figure 4A The user interface 41 shown in the example can be referred to as the home screen.

[0148] like Figure 4A and Figure 4B As shown, after the source-side device 100 detects a swipe operation from the top of the display screen, in response to the swipe operation, the source-side device 100 can display the following on the user interface 41: Figure 4B Window 111 is shown. (As shown) Figure 4B As shown, window 111 may display control 111A, which can accept user operations (such as touch operations, click operations) to turn the mirroring function of the source device 100 on / off. The display form of control 111A may include icons and / or text (such as the text "Mirroring", "Wireless Mirroring", "Multi-screen Interaction", "Large Screen Projection", etc.). Window 111 may also display on / off controls for other functions such as Wi-Fi, Bluetooth, flashlight, etc.

[0149] like Figure 4B As shown, the source device 100 can detect a click operation on the control 111A and enable the screen mirroring function. In some embodiments, after detecting a user operation on the control 111A, the source device 100 can change the display form of the control 111A, such as adding a shadow when displaying the control 111A.

[0150] Not limited to Figure 4A On the user interface 41 shown, the user can also input a downward swipe operation on other interfaces of the settings application or the user interface of other applications to trigger the source device 100 to display window 111.

[0151] Not limited to Figure 4A and Figure 4B The illustration shows a user operation performed on control 111A within window 111. Optionally, the user operation to enable the screen mirroring function can also be an operation to enable a function option in the settings application. For example, the user can also bring the source device 100 close to the NFC tag of the end device 200 to trigger the source device 100 to enable the screen mirroring function. This application embodiment does not limit the user operation for enabling the screen mirroring function.

[0152] In response to a user's operation to enable the screen mirroring function, the source device 100 enables one or more of the following wireless communication technologies in the wireless communication module 160: WLAN, Bluetooth, or NFC. It can also discover screen mirroring devices that can establish a screen mirroring communication connection through one or more of the following wireless communication technologies: Wi-Fi, Bluetooth, and NFC.

[0153] After the source-side device 100 locates a screen-sharing device with which a communication connection can be established, it can display the identifiers of these screen-sharing devices. For example, it can display identifiers such as... Figure 4C The window shown is 112.

[0154] like Figure 4CAs shown, window 112 can display the identifiers and connection options of one or more screen-castable devices. The source device 100 can detect user actions applied to these connection options and establish a communication connection with the screen-castable device indicated by the device identifier corresponding to that option. These one or more screen-castable device identifiers and connection options include identifier 112A and connection option 112B. When the source device 100 detects a user action applied to connection option 112B, in response to this action, the source device 100 can send a communication connection to the device identified as "HUAWEI20" displayed in identifier 112A. Furthermore, connection option 112B can be updated as follows... Figure 4D Option 112C, as shown, is used to prompt the user that the source device 100 is searching for devices that can be used for screen mirroring.

[0155] It is understood that the embodiments of this application do not limit the user operation of the source device 100 in selecting the device to establish a communication connection. In addition to displaying the identifier of the screen-castable device, the source device 100 can also display other information, such as the device type of the screen-castable device, etc. The embodiments of this application do not limit this.

[0156] After receiving a communication connection request from the source device 100, the device identified as "HUAWEI 20" can display the following: Figure 4E The user interface 42 shown includes a window 201, which prompts the user whether they agree to establish a communication connection. Window 201 may include a confirmation control 201A and a cancellation control 201B. The confirmation control 201A, in response to the user's action, establishes a communication connection with the source device 100. In this case, the device identified as "HUAWEI20" is the end device 200 that has established a communication connection with the source device 100. The end device 200 can display the user interface provided by the source device 100, as detailed in the subsequent description of the user interface displayed on the end device 200. The cancellation control 201B, in response to the user's action, refuses to establish a communication connection with the source device 100.

[0157] Optionally, in some embodiments, the end-side device 200 may not display a prompt message after receiving a communication connection request, i.e., it may not display a prompt message such as... Figure 4E The window 201 shown directly establishes a communication connection with the source device 100.

[0158] In some embodiments, both the source device 100 and the end device 200 can run a screen mirroring application to support the establishment of a communication connection between the two devices and the subsequent screen mirroring process.

[0159] In this embodiment, the communication connection established between the source device 100 and the end device 200 can be a Wi-Fi connection, Bluetooth connection, NFC connection, or remote connection, or it can be a wired connection such as a data cable connection. This embodiment does not impose any restrictions on this.

[0160] S102-S110, screen mirroring process.

[0161] S102, after establishing a communication connection with the end-side device 200, the source-side device 100 captures the screen content according to the projection frame rate, obtains the image frame, and sends the image frame to the end-side device 200.

[0162] After the source device 100 and the terminal device 200 establish a communication connection, during screen mirroring, the projection frame rate of the source device 100 is determined by the speed at which the image displayed on its screen changes. The faster the image changes on the source device 100's screen, the higher the projection frame rate determined by the source device 100. In other words, the faster the image changes on the source device 100, the more screen content it will capture and send to the terminal device 200. This ensures that the image changes on the source device 100 are captured, and the image frames received by the terminal device 200 reflect this change process, preventing abrupt changes in the image and avoiding any stuttering or lag for the user.

[0163] After the source device 100 and the terminal device 200 establish a communication connection, the screen displayed on the source device 100 and the changes to that screen are all controlled by the user. The user can control the source device 100 to perform any type of operation or function, and the source device 100 can display different content on the screen in response to user input. For example, the source device 100 can launch a social networking application and display social content in response to user input. Or, it can launch a reading application and display novel text in response to user input. Or, it can launch a social networking application in response to user input, and so on.

[0164] It can be said that the user's input to the source device 100 to change the content displayed on the source device 100 determines the screen projection frame rate of the source device 100.

[0165] In some embodiments, the projection frame rate of the source device 100 does not exceed the full frame rate, i.e., the maximum frame rate. The full frame rate can be preset by the source device 100, for example, it can be 60 FPS, etc., which is not limited here.

[0166] The source-side device 100 can continuously capture the screen content displayed on the display screen according to a determined projection frame rate, obtain the corresponding image frames, and send the image frames to the end-side device 200 based on the communication connection between the source-side device 100 and the end-side device 200. In some embodiments, the source-side device 100 can send one image frame to the end-side device 200 after capturing one image frame, that is, continuously send image data streams to the end-side device 200 during the screen mirroring process. Essentially, S102 will be performed multiple times during the screen mirroring process.

[0167] In some implementations, the source-side device 100 can timestamp each image frame according to the capture time, or number it according to the order in which it was captured.

[0168] In some implementations, the source-side device 100 can encode the captured image frames and send them to the end-side device 200. If the source-side device 100 has insufficient computing power or for other reasons, the encoding stage may take a long time.

[0169] S103, during the execution of S102, the source device 100 synchronizes the projection frame rate to the end device 200.

[0170] In this embodiment of the application, the source device 100 can periodically synchronize the projection frame rate to the end device 200 based on the communication connection between the source device 100 and the end device 200, or synchronize the new projection frame rate to the end device 200 when the projection frame rate changes.

[0171] In other words, S103 will execute multiple times during the screen mirroring process.

[0172] The source device 100 can send the projection frame rate and the image frames captured in S102 together to the end device 200, or they can be sent separately. This application embodiment does not limit this.

[0173] S104, the source-side device 100 starts the first application and displays the first user interface on the display screen.

[0174] In this embodiment, the order of S101 and S104 is not limited. The source device 100 may execute S101 first and then execute S104 during the execution of S102, or it may execute S104 first and then execute S101 and S102. Figure 3 The flowchart shown is illustrated using the previous execution order as an example.

[0175] In this application embodiment, the first application is an application that can slide or scroll to display content in the user interface in response to user operation, such as a browser, social application, reading application, etc.

[0176] Figure 5A and Figure 5B An example is shown of the process by which the source-side device 100 starts the first application.

[0177] Figure 5A This is the user interface 41 displayed by the source-side device 100, which can be the main interface provided by the source-side device 100. For example... Figure 5A As shown, the source device 100 can detect user actions (such as clicks, touches, etc.) applied to the social application icon 501 on the main interface. The source device 100 can respond to this user action by displaying... Figure 5B The user interface 51 provided by this social application is shown. This user interface 51 is an example of a first user interface.

[0178] Not limited to Figure 5A In some other embodiments, the source-side device 100 can launch the first application in other ways, as shown in the method for launching the first application. For example, the source-side device 100 can also launch the first application in response to a voice command. Another example is that the source-side device 100 can be used with a mouse, and can launch the first application in response to a double-click operation received on the mouse after the mouse cursor is positioned at the location of the first application icon, and so on.

[0179] In some embodiments, the first user interface may include system content and page content.

[0180] System content refers to the content provided by the system application when the source device 100 is running the system application, such as the status bar and system navigation bar. System content is usually displayed in a fixed area on the screen, and the electronic device does not change the position of the displayed system content based on user operations.

[0181] Page content refers to the content provided by the first application currently running in the foreground on the source device 100, such as the application title bar, application menu bar, application internal navigation bar, application content, etc. When the source device 100 launches the first application, it can load the page content provided by the first application via the network or locally. Furthermore, the first user interface only displays a portion of the page content. When the source device 100 receives a user's swipe gesture on the screen, the remaining portion of the page content can be displayed in the first user interface. In other words, if the page content provided by the first application is long, only a portion of the page content is typically displayed on the screen. The source device 100 can detect user swipe gestures on the page content, and in response to these gestures, it scrolls the scrollable area of ​​the page content in the first user interface. This allows users to browse more page content according to their needs.

[0182] The page content can be the main page of the first application or other pages provided by the first application; there are no restrictions here. The page content can come from the local device 100 on the source side or from the network.

[0183] For example, refer to Figure 5B The user interface 51 shown includes a status bar containing system content, and other content outside the status bar containing page content provided by social applications. The status bar is located in area 502 of the display screen, and the other content is located in area 503 of the display screen.

[0184] In other embodiments, the first user interface may also consist only of page content. For example, Figure 5B The user interface 51 shown can also include only content other than the status bar, that is, only the content in area 503.

[0185] Based on whether the content can be scrolled in response to user operation, the display area where the first user interface is located, i.e. the display screen of the source device 100, can be divided into two parts: a scrollable area and a non-scrollable area.

[0186] 1. Scrollable area

[0187] A scrollable area is an area that changes its scrolling content in response to user actions (such as swiping up or down). The content displayed in a scrollable area can scroll in response to user actions (such as swiping up or down). Content displayed in a scrollable area can include social media content, news, fiction text, images, and more.

[0188] For example, refer to Figure 5B The user interface 51 shown has a scrollable area 503a within area 503, where multiple items are displayed.

[0189] 2. Non-scrollable areas

[0190] A non-scrollable area is an area that does not scroll in response to user actions (such as swiping up or down). The content displayed in a non-scrollable area will not scroll in response to user actions (such as swiping up or down). Content displayed in a non-scrollable area may include system content, and portions of the page content such as the menu bar, search bar, application navigation bar, etc.

[0191] For example, refer to Figure 5B The user interface 51 shown, the status bar area 502, and a portion of area 503b within area 503 are all non-scrollable areas. The menu bar is displayed in the non-scrollable area 503b. Figure 5BWhen different options in the middle menu bar are selected, different content can be displayed in the scrollable area 503a.

[0192] In other embodiments of this application, the display area where the first user interface is located may only include scrollable areas and exclude non-scrollable areas. For example, Figure 5B The user interface 51 shown can display only the page content displayed in the scrollable area 503a.

[0193] The content of the first user interface depends on the display mechanism of the source device 100 and the content provided by the first application. The positions of the scrollable and non-scrollable areas on the display screen depend on the positions of the various contents in the first user interface on the display screen.

[0194] In optional step S105, the source device 100 synchronizes the application information of the first application to the end device 200.

[0195] In this embodiment, the source-side device 100 can periodically synchronize application information of the applications running on the source-side device 100 to the end-side device 200 based on the communication connection between the source-side device 100 and the end-side device 200. Alternatively, it can synchronize application information of the newly running applications to the end-side device 200 when the running applications change. The application can be the first application or other applications.

[0196] The source-side device 100 can send the application information of the application and the image frame captured in S102 together to the end-side device 200, or they can be sent separately. This application embodiment does not limit this.

[0197] Application information may include one or more of the following: the application's identifier (such as name, code), the application type (such as browser, social application, reading application, etc.), and information about the user interface provided by the application. User interface information may include, for example, the type of user interface, the position and size of scrollable areas within the user interface, the position and size of non-scrollable areas, etc.

[0198] S106, during the execution of S102, the source-side device 100 scrolls the page content located in the scrollable area of ​​the first user interface at a speed greater than a first value.

[0199] The source-side device 100 can execute S106 after executing S101, during the execution of S102.

[0200] In some embodiments, the source-side device 100 may receive a first operation input by the user on the source-side device 100 and execute S106. The first operation may be a sliding operation detected by the source-side device 100 acting on a scrollable area in the display screen (e.g., a sliding gesture in any direction, such as an upward sliding gesture or a downward sliding gesture), or a sliding operation received on the mouse after the mouse cursor is located in the scrollable area when the source-side device 100 is used with a mouse, or a voice command, etc.

[0201] The source-side device 100 scrolling the page content in a scrollable area means that the page content in the scrollable area scrolls or moves in a certain direction at a certain speed. During this process, some page content is moved out of the scrollable area and is no longer displayed, some page content changes its position in the scrollable area, and new page content appears in the scrollable area.

[0202] The direction in which the source-side device 100 scrolls the page content is related to the first operation. If the first operation is a sliding operation on a scrollable area, the direction in which the source-side device 100 scrolls the page content can be the same as the direction of the sliding operation. For example, if the first operation is a sliding operation in any direction, the source-side device 100 can also scroll the page content in any direction. In some other embodiments, the source-side device 100 is pre-set to only scroll the page content in a fixed direction (such as an upward or downward direction). When the source-side device 100 receives a movement operation containing a motion vector in that fixed direction, it can scroll the page content in that fixed direction.

[0203] For example, the direction in which the page content in the scrollable area is scrolled by the source-side device 100 can be upward, downward, etc. Upward scrolling refers to the page content displayed on the screen moving from the bottom to the top of the screen. Similarly, downward scrolling refers to the page content displayed on the screen moving from the top to the bottom of the screen.

[0204] The speed at which the source-side device 100 scrolls the page content is related to the first operation. If the first operation is a swipe operation on a scrollable area, the speed at which the source-side device 100 scrolls the page content is related to the speed of that swipe operation. Specifically, when the source-side device 100 receives a swipe operation on a scrollable area, it scrolls the page content in the scrollable area in the scrolling direction at the speed of that swipe operation. At this time, from the user's perspective, the page content in the scrollable area moves following the swipe operation of the hand. After the user ends the swipe operation, the source-side device 100 continues to scroll the page content in the scrollable area in the scrolling direction due to inertia. In some embodiments, the source-side device 100 can gradually reduce the scrolling speed of the page content after the user ends the swipe operation until it stops scrolling. In other embodiments, the source-side device 100 can first increase the scrolling speed of the page content after the user ends the swipe operation, and then gradually decrease the scrolling speed of the page content until it stops scrolling. It can be seen that during the process of the source-side device 100 scrolling the page content located in the scrollable area of ​​the first user interface, the scrolling speed changes with the first operation.

[0205] In some implementations, when the first operation is a sliding operation acting on a scrollable area, the motion vector V of the sliding operation on the display screen can be calculated using the following formula:

[0206]

[0207] Among them, Xv and Yv 2 These are the motion vectors of the sliding operation in the X and Y directions, respectively. The X and Y directions can be the direction from the left side of the display screen to the right side and the direction from the bottom of the display screen to the top side, respectively.

[0208] When the first operation meets certain conditions, such as the speed of the swipe operation being greater than a certain value, the source device 100 will scroll and display the page content located in the scrollable area of ​​the first user interface at a faster speed (such as a speed greater than the first value). The first value can be preset and is not limited here.

[0209] As described in S102 above, when the source device 100 scrolls the page content in the scrollable area of ​​the first user interface at a relatively fast speed, the speed of change of the screen displayed is relatively fast, and the projection frame rate determined by the source device 100 is also relatively high. Therefore, during the execution of S106, the source device 100 will use this higher projection frame rate to capture the screen content, obtain multiple image frames, and send the image frames to the end device 200.

[0210] Figures 5B-5EAn exemplary scenario is shown where the source-side device 100 scrolls to display page content located in a scrollable area of ​​a first user interface.

[0211] like Figures 5B-5C As shown, the source-side device 100 is in Figure 5B An upward sliding operation acting on the scrollable area 503a was detected in the middle. Figure 5C Upon detecting that the user has finished the upward swipe operation, the source device 100 responds by scrolling the page content in the scrollable area 503a upwards. During this process, the page content in the scrollable area 503a can follow the user's hand movement; the area of ​​the display screen that the user's hand touches during the movement displays the same content, for example... Figure 5B and Figure 5C The animal pictures are at the bottom of the page.

[0212] like Figures 5C-5E As shown, after the user finishes the above upward swipe operation, the source device 100 continues to scroll upward to display the page content in the scrollable area 503a due to inertia.

[0213] Figures 5B-5E The illustration only shows the content displayed on the screen when the source-side device 100 scrolls through the page content. In a real implementation, the source-side device 100 may display more images during the scrolling process.

[0214] The source-side device 100 uses a determined projection frame rate to capture image frames of screen content, which may include... Figures 5B-5E The four image frames shown are from the user interface. In subsequent embodiments, the images captured by the source-side device 100 will be... Figures 5B-5E The four image frames shown are referred to as image frame 1, image frame 2, image frame 3, and image frame 4, respectively.

[0215] In other embodiments, the endpoint device 200 may receive a second operation input by the user on the endpoint device 200 during the display of the projected content, and send the operation information of the second operation to the source device 100 to trigger the source device 100 to execute S106. In other words, during the mirroring process, the user can control the content displayed by the source device 100 on the endpoint device 200.

[0216] The second operation can be a sliding operation detected by the end device 200 on the mirrored projection area of ​​the display screen used to display the projection content (such as a sliding gesture in any direction, such as an upward sliding gesture or a downward sliding gesture), or a sliding operation received on the mouse after the mouse cursor is in the scrollable area when the end device 200 is used with a mouse, or a click operation received on the remote control after the scrollable area is selected when the end device 200 is used with a remote control, or a voice command, etc.

[0217] The way the source device 100 scrolls and displays the page content of the scrollable area triggered by the second operation detected by the end device 200 (such as speed, direction lights) can be referred to the way the source device 100 scrolls and displays the page content of the scrollable area triggered by the first operation detected by the source device 100 mentioned above, and will not be repeated here.

[0218] In optional step S107, the source-side device 100 synchronizes the operation information of the first operation to the end-side device 200.

[0219] In this embodiment, the source-side device 100 can synchronize detected operation information to the end-side device 200 based on the communication connection between the source-side device 100 and the end-side device 200. This operation may include a first operation or other operations.

[0220] The source-side device 100 can send the operation information and the image frame captured in S102 together to the end-side device 200, or they can be sent separately. This application embodiment does not limit this.

[0221] Operation information may include any one or more of the following: operation type (e.g., sliding operation type), operation direction, operation speed, operation duration, operation trajectory, and operation motion vector.

[0222] S108, the end-side device 200 receives the image frame sent by the source-side device 100 and sends the received image frame into the display queue.

[0223] During screen mirroring, high-quality communication between the source device 100 and the end device 200 may not be consistently guaranteed. Therefore, some image frames sent by the source device 100 to the end device 200 may be lost during communication, and some image frames sent by the source device 100 may not be received by the end device 200. For example... Figures 5B-5E Of the four image frames captured by the source device 100, the end device 200 may only receive image frame 1 and image frame 2, while image frame 3 and image frame 4 are lost due to communication issues.

[0224] In some embodiments, the edge device 200 can receive encoded image frames and decode them to obtain image frames. If the edge device 200 has insufficient computing power or for other reasons, it may take a long time to decode and obtain image frames.

[0225] In this embodiment, the end-side device 200 can use any of the following strategies to send the received image frames into the display queue: Strategy 1: The end-side device 200 sends the image frames into the display queue sequentially according to the order in which they are received. Strategy 2: If the source-side device 200 has added a timestamp to the image frame, the end-side device 200 can send the image frames into the display queue sequentially according to the time indicated by the timestamp. Strategy 3: If the source-side device 200 has added a sequence number to the image frame, the end-side device 200 can send the image frames into the display queue sequentially according to the sequence number.

[0226] In some implementations, to ensure low latency during screen mirroring, the endpoint device 200 can discard received outdated image frames to avoid sending them to the display queue, or send them to the display queue and then retrieve them. Outdated image frames refer to those received or decoded by the endpoint device 200 whose capture time from the source device 100 is greater than a first duration. The endpoint device 200 can discard some of the earlier image frames (e.g., two) in the display queue when the number of image frames in the queue exceeds a certain amount (e.g., two). The endpoint device 200 can also discard image frames whose capture time is greater than the reception time or decoding time based on the timestamp carried in the image frame. In this way, by discarding outdated image frames, the endpoint device 200 can ensure low latency during screen mirroring, giving the user the impression that the endpoint device 200 and the source device 100 are displaying images synchronously, thus providing a better user experience.

[0227] In this embodiment, the display queue is a queue in the edge device 200 used to provide display images to the display screen. Images in the display queue are provided to the display screen sequentially according to the order in which they are entered into the queue. Image frames that have already been provided to the display screen are no longer stored in the display queue. The display queue can have a predetermined size, for example, it can store up to four image frames. Therefore, the image frames contained in the display queue are updated in real time, and the image frames contained in the display queue can be different at different times. Simultaneously, the edge device 200 can also be configured with a buffer, in which image frames that have been provided to the display screen by the display queue can be stored.

[0228] Typically, the endpoint device 200 follows a first-in, first-out (FIFO) rule, retrieving the first image frame sent to the display queue at a fixed frequency for display. This fixed frequency is related to the projection frame rate of the source device 100; for example, the fixed frequency = 1 / projection frame rate. For instance, if the projection frame rate is 60 frames per second, the fixed frequency could be 16.66 milliseconds. In specific implementations, the endpoint device 200 can continuously generate synchronization signals at this fixed frequency, retrieving an image frame from the display queue for display when the synchronization signal arrives. The time point at which the synchronization signal is generated will be referred to as the synchronization time point in subsequent embodiments.

[0229] refer to Figure 6 , Figure 6 An example is shown of image frames included in the display queue of the end-side device 200 at different times over a period of time.

[0230] like Figure 6 As shown, the horizontal axis represents the timeline, on which synchronization signals are generated at the projection frame rate. The synchronization time point for generating the synchronization signal is the time when the terminal device 200 retrieves an image frame from the display queue for display. The current time point is located between synchronization time points 2 and 3, and the time point after the current time point represents a future time that has not yet arrived.

[0231] Figure 6 The display queues for four different time periods are shown. The time periods corresponding to the display queues can be found on the time axis below them.

[0232] Initially, the display queue contains an image frame a, which is provided to the display screen of the end device 200 for display when the synchronization signal 1 arrives.

[0233] Subsequently, a new image frame b is added to the display queue, and this image frame b is provided to the display screen of the end device 200 for display at the arrival time of the synchronization signal 2.

[0234] Then, the end device 200 receives image frame c and image frame d in sequence. Image frame c and image frame d are received in the display queue in sequence. Image frame c is provided to the display screen of the end device 200 for display when the synchronization signal 3 arrives, and image frame d is provided to the display screen of the end device 200 for display when the synchronization signal 4 arrives.

[0235] Subsequently, the end-side device 200 received image frame eg in sequence. Image frames eg were received in the display queue in order. Image frames e and f were removed from the display queue because they were outdated. Image frame g was provided to the display screen of the end-side device 200 for display when the synchronization signal 7 arrived.

[0236] During the generation time of synchronization signal 5-synchronization signal 6, there are no image frames in the display queue.

[0237] S109, the end-side device 200 predicts the predicted image frame corresponding to the first time point based on the latest image frame entering the display queue.

[0238] End-side device 200 executes S109 in any of the following scenarios:

[0239] In scenario 1, after the end device 200 establishes a communication connection with the source device 100 for screen mirroring, it executes S109.

[0240] Scenario 2: After the end device 200 establishes a communication connection with the source device 100 for screen mirroring, if the communication quality between the two parties is lower than the threshold, S109 is executed.

[0241] The communication quality between the end-side device 200 and the source-side device 100 can be measured by parameters such as communication signal strength, communication delay, and signal-to-noise ratio.

[0242] If the communication quality between the end device 200 and the source device 100 is lower than a certain value, it indicates that the communication quality between the two is poor. In this case, it is highly likely that the image frames sent by the source device 100 to the end device 200 will be lost during the communication process due to the poor communication quality. In other words, scenario 2 is very likely to cause frame loss. Executing S109 in scenario 2 can effectively avoid the poor user experience caused by possible frame loss.

[0243] Scenario 3: After the terminal device 200 establishes a communication connection with the source device 100 for screen mirroring, if the source device 100 is running a specified type of application in the foreground and the source device 100 executes S106, that is, scrolling the page content in the scrollable area of ​​the first user interface at a speed greater than the first value, then the terminal device 200 executes S109.

[0244] The endpoint device 200 can determine whether the source device 100 is running a specified type of application in the foreground based on the application information synchronized by the source device 100 in step S105 above. The specified type of application is an application that displays content in the user interface in response to user operations by sliding or scrolling, such as a browser, social application, reading application, etc.

[0245] The endpoint device 200 can also determine whether the source device 100 has received a specified type of operation based on the operation information synchronized by the source device 100 in step S107 above. Alternatively, the endpoint device 200 can also determine whether it has received a specified type of operation. The specified type of operation refers to an operation that triggers the source device 100 to scroll and display the page content located in the scrollable area of ​​the first user interface at a speed greater than a first value, such as a swipe operation with a speed greater than a certain value, etc.

[0246] If the source device 100 is running a specified type of application in the foreground and executes S106, the source device 100 will perform screen mirroring at a higher screen mirroring frame rate. If frame drops occur at a high screen mirroring frame rate, it will cause very noticeable visual stuttering for users. Therefore, executing S109 in scenario 3 can effectively avoid the poor user experience caused by frame drops at a high screen mirroring frame rate.

[0247] Based on the aforementioned steps, after the source device 100 executes S104 and S106, that is, after the source device 100 starts the first application and scrolls the page content located in the scrollable area of ​​the first user interface at a speed greater than the first value, the end device 200 can execute S109 during the screen mirroring process.

[0248] Scenario 4: When the difference between the display frame rate of the terminal device 200 and the projection frame rate of the source device 100 is greater than a certain value, such as the second value, the terminal device 200 executes S109.

[0249] The display frame rate of the endpoint device 200 is determined by the number of image frames displayed on the screen per unit time during the mirroring process. For example, if the endpoint device 200 follows... Figure 6 The display queue shown is used to display the projected content. Assuming a synchronization signal is generated every 16.66 milliseconds, the display frequency is 60 FPS during the generation time of synchronization signals 1-4, and 0 FPS during the generation time of synchronization signals 4-6.

[0250] The closer the display frame rate of the endpoint device 200 is to the projection frame rate of the source device 100, the better the mirrored projection effect will be for the user. The greater the difference between the display frame rate of the endpoint device 200 and the projection frame rate of the source device 100, the more severe the frame drop problem will be during the mirrored projection process. Executing S109 in scenario 4 can prevent the user experience from being negatively impacted by the subsequent continuous frame drop problem when it begins to appear.

[0251] In addition to the scenarios listed above, in other embodiments of this application, the end-side device 200 may also combine any number of the scenarios described above to execute S109. For example, the end-side device 200 may execute S109 while simultaneously satisfying scenario 3 and scenario 4.

[0252] When the end-side device 200 executes S109, it can employ the following strategies:

[0253] Strategy 1: When the end device 200 receives a new image frame, it executes S109 once.

[0254] Strategy 2: S109 is executed once for each new image frame added to the display queue of the end device 200.

[0255] Strategy 3: End-side device 200 periodically executes S109.

[0256] In other words, in this embodiment of the application, S109 will be executed multiple times.

[0257] The first time point is located after the current execution time of S109. The first time point may include one or more synchronous time points where the display frame rate of the terminal device 200 is lower than the projection frame rate when the terminal device 200 displays images only according to the image frames in the current display queue. In other words, the first time point includes one or more synchronous time points after all the image frames in the current display queue have been displayed, assuming the terminal device 200 displays images according to the image frames in the current display queue. The number of these multiple synchronous time points can be a fixed number, which can be preset by the terminal device 200 or set by the user, for example, it can be 2.

[0258] For example, refer to Figure 6 Assuming that the terminal device 200 executes S109 at the current time point, it displays images according to the image frames in the display queue at the current time point. At synchronization time point 4, all image frames in the display queue have been displayed, and after synchronization time point 4, the display frame rate of the terminal device 200 is 0. Therefore, one or more synchronization time points after synchronization time point 4 are considered first time points. For example, synchronization time points 5 and 6 can be considered first time points.

[0259] The following describes how the end-side device 200 uses the latest image frame entering the display queue to predict the predicted image frame corresponding to the first time point.

[0260] The strategy by which the edge device 200 sends received image frames into the display queue can be found in the relevant description in S108. The most recently entered image frame into the display queue can also be found in the relevant description in S108. Clearly, the most recently entered image frame into the display queue is the most recently received image frame by the edge device 200.

[0261] During execution of S109, the latest image frame entering the display queue may have already been partially or entirely sent to the display screen and removed from the display queue. Therefore, the end device 200 can retrieve the latest image frame entering the display queue from the display queue and / or the buffer.

[0262] For example, refer to Figure 6 Assuming that the terminal device 200 executes S109 at the current time, then based on the current time, the two most recently sent image frames into the display queue are image frame c and image frame d. The terminal device 200 can obtain image frame c and image frame d from the current display queue.

[0263] In some embodiments, the endpoint device 200 can use the two most recently entered image frames in the display queue to predict the predicted image frame corresponding to the first first time point. According to the strategy of the endpoint device 200 in S108 above for sending received image frames into the display queue, the two most recently entered image frames in the display queue may be two adjacent image frames captured by the source device 100, or they may be two non-adjacent image frames captured by the source device 100. That is, the image frame most recently captured by the source device 100 before capturing image frame 2 may be image frame 1, or it may be another image frame.

[0264] The following is based on Figure 6 Taking image frame c and image frame d as examples from the mid-end device 200, the two most recently entered image frames in the display queue, image frame c entered the display queue before image frame d. Let's assume image frame c is specifically... Figure 5B Image frame 1 is shown, and image frame d is specifically... Figure 5B Image frame 2 is shown.

[0265] The process by which the edge device 200 predicts the first predicted image frame corresponding to the first time point based on the two most recently entered image frames in the display queue may include the following steps:

[0266] S1091, the end-side device 200 compares image frame 1 and image frame 2 to determine the motion region in image frame 2.

[0267] The motion region in image frame d refers to the area in image frame d where the displayed content has changed compared to image frame c. It is equivalent to the area where the content of the scrollable area of ​​the user interface provided by the source device 100 in image frame d is displayed.

[0268] In some embodiments, S1091 performed by the end-side device 200 specifically includes the following steps:

[0269] Step 1: Traverse all pixels of image frame 2, determine the change value of each pixel in image frame 2 at the same position compared to image frame 1, and then determine the pixels whose pixel value changes exceed the threshold T to obtain region 1 composed of these pixels.

[0270] Essentially, the end-side device 200 binarizes the differences between image frame 1 and image frame 2 during the motion process to identify the region of motion.

[0271] The degree of change of each pixel in image frame 2 can be calculated in the following way:

[0272] D k (x,y)=|f k (x,y)-f k-1 (x,y)|

[0273] Among them, f k-1 (x,y),f k (x, y) represent the pixel values ​​of the pixel with coordinates (x, y) in image frame 1 and image frame 2, respectively. k (x,y) represents the change in the pixel at coordinates (x,y) in image frame 2 compared to image frame 1.

[0274] The threshold T can be preset, for example, it can be dynamically obtained by calculating it in advance using the Otsu's method. T can also be an empirical value.

[0275] Region 1, as determined in step 1, may not be a standard shape (such as a rectangle) and may be a discrete region. However, based on the motion of image frames 1 and 2 captured by the source device 100, the scrolling area is usually a standard shape and a concentrated area when the source device 100 scrolls the page content in the user interface.

[0276] For example, comparing image frame 1 and image frame 2, the actual scrolling area should be... Figure 5C The scrollable area 503a in the image frame 1 and the image frame 2 may be a non-standard and discrete area in the actual scrollable area 503a because there are some blank spaces or the same content in the image frame 1 and the image frame 2.

[0277] In order to obtain a more accurate motion region in image frame 2, after S1091, the end device 200 can also perform further correction operations based on region 1.

[0278] Step 2: Correct region 1 to obtain region 2.

[0279] In some implementations, the edge device 200 can standardize the shape of region 1, for example, by obtaining a standardized rectangle based on the shape of region 1. Specifically, the edge device 200 can obtain the maximum x-coordinate of each pixel in region 1. max Minimum x-coordinate min Maximum y-coordinate max Minimum y-coordinate min Then determine the following four coordinate points: (x min ,y min ),(x min ,y max ),(x max ,y min ),(x max ,y max The area formed by the above four coordinate points is defined as a standard area.

[0280] In some implementations, the end-side device 200 can dediscretize the shape of region 1, for example, by obtaining a concentrated region based on the shape of region 1.

[0281] The edge device 200 can combine the motion regions determined during the previous prediction of image frames and the union of region 1 over a period of time to obtain region 2. If the edge device 200 has not predicted image frames in the previous period of time, the edge device 200 can use two or more image frames that entered the display queue before image frame 1 and image frame 2 to perform one or more operations similar to step 1 above to obtain one or more regions, and then take the union of these regions with region 1. Since the user's input on the source device 100 is usually the same over a period of time, the scrollable area in the source device 100 will not change. Therefore, a more accurate motion region can be obtained by taking the union as described above.

[0282] The endpoint device 200 can combine the user's sliding operation habits with those of the source device 100 to obtain a concentrated area based on the shape of region 1. For example, if the source device 100 receives a vertical sliding operation, the endpoint device 200 can directly extend the width of region 1 to the width of image frame 2. Similarly, if the source device 100 receives a horizontal sliding operation, the endpoint device 200 can directly extend the length of region 1 to the length of image frame 2. This approach, taking into account both the user's operation and the way the source device 100 responds to the operation, can obtain a more accurate motion area.

[0283] The region obtained by combining any one or more of the above three methods can be called region 2. Region 2 is the motion region in image frame 2.

[0284] S1092, End-side device 200 compares image frame 1 and image frame 2, and determines the motion vector in image frame 2.

[0285] When displaying image frame 1, the source-side device 100 moves the page content in the motion area according to the motion vector. After the movement, image frame 2 is displayed. Movement vector. This indicates the distance and direction of movement. In other words, comparing image frame 1 and image frame 2, a certain displayed content or target point in image frame 1 moves according to the movement vector. After the movement, it is displayed in the motion area of ​​image frame 2. In other words, when the source-side device 100 displays image frame 1 and image frame 2 on the display screen, the movement vector when the same content moves from its position in image frame 1 to its position in image frame 2 is this movement vector.

[0286] In other words, assuming the image f1(x,y) passes through the vector The image f2(x,y) is obtained by translation, which means we need to obtain...

[0287] f2(x,y)=f1(x-dx,y-dy)

[0288] Mapped to the frequency domain, i.e.

[0289] F2(u,v)=F1(u,v)*e -i*2π*(u*dx+v*dy)

[0290] Cross power spectrum

[0291]

[0292] The inverse Fourier transform yields the impulse function; by taking the peak value of the impulse function, the offset vector can be obtained.

[0293] Image f1(x,y) represents image frame 1, and image f2(x,y) represents image frame 2.

[0294] In other embodiments, the end-side device 200 may also use other methods to calculate the movement vector. For example, the end-side device 200 can also calibrate marker pixels that display the same content in image frame 1 and image frame 2, calculate the movement vector of the marker pixel from its position in image frame 1 to its position in image frame 2, and use this as the movement vector.

[0295] As can be seen, S1091-S1092 above determine the motion region and movement vector of image frame 2 relative to the previous frame image (i.e., image frame 1) in the display queue by comparing the latest image frame 2 sent into the display queue with image frame 1.

[0296] S1093, the end-side device 200, based on image frame 2, displays the content in the motion area according to the motion vector. The image is moved, and the empty area in the moving area is filled with the predicted data to obtain the predicted image frame; the predicted data is obtained based on the content displayed in the empty area of ​​image frame 2.

[0297] Specifically, based on image frame 2, the edge device 200 displays the content in the motion area according to the motion vector. When a content is moved, some of the content that was originally in the moving area will be moved out of the moving area and will no longer be displayed. An empty area will also appear in the original moving area. This empty area and the area originally occupied by the moved content can be the same size or different.

[0298] Then, the end device 200 obtains the prediction data based on the content originally displayed in the empty area of ​​the image frame 2, and fills the empty area with the prediction data.

[0299] In some embodiments, the end-side device 200 may employ image processing methods such as mean blur, median blur, Gaussian blur, bilateral blur, surface blur, box blur, double blur, bokeh blur, tilt-shift blur, aperture blur, grainy blur, radial blur, and directional blur to process the content originally displayed in the empty area of ​​image frame 2, thereby obtaining predicted data. The predicted data obtained through blurring image processing methods has lower clarity compared to the content originally displayed in the empty area of ​​image frame 2.

[0300] In other embodiments, the end device 200 may also directly regard the content originally displayed in the empty area of ​​the image frame 2 as prediction data.

[0301] In some other embodiments, the edge device 200 may also use a neural network algorithm to perform image prediction based on the content originally displayed in the empty area of ​​image frame 2, thereby obtaining prediction data. This neural network algorithm can be trained using a large amount of content displayed on the user interface of an electronic device as input and the content displayed after scrolling through the content on the user interface as output. For example, if the content originally displayed in the empty area of ​​image frame 2 is part of a typical pattern, the edge device 200 can predict another part of that typical pattern using the algorithm.

[0302] In some other embodiments, the end-side device 200 may also store image frames sent by the source-side device 100 over a period of time, for example, in a cache. The end-side device 200 can obtain prediction data based on the stored image frames. For example, if a user inputs multiple user operations (e.g., repeated up and down swiping operations) over a period of time, triggering the source-side device 100 to repeatedly display the same image frames, the end-side device 200 can obtain prediction data based on the previously displayed image frames.

[0303] The methods mentioned above for obtaining forecast data can also be combined in any way.

[0304] In other embodiments, the edge device 200 can use the most recently entered image frame in the display queue to predict the prediction image frame corresponding to the first first time point. The following uses... Figure 6 Let's take image frame d as an example, the latest image frame to enter the display queue of the mid-end device 200. Figure 5B Image frame 2 is shown.

[0305] The process by which the edge device 200 predicts the first predicted image frame corresponding to the first time point based on the latest image frame entering the display queue may include the following steps:

[0306] S1091', the end-side device 200 determines the motion region in image frame 2 based on the application information of the first application.

[0307] Here, the moving region in image frame 2 refers to the area in image frame 2 where the displayed content has changed compared to the most recently captured image frame (e.g., image frame 1) before image frame 2 by the source device 100. This is equivalent to the area where the content of image frame 2 is displayed in the scrollable area of ​​the user interface provided by the source device 100.

[0308] If the source device 100 executes S105, the end device 200 can determine the position and size of the scrollable area in the first user interface based on the information of the first user interface contained in the application information of the first application sent by the source device 100, and determine the area where the content of the scrollable area is located when the image frame 2 is displayed in the first user interface as the motion area of ​​the image frame 2.

[0309] S1092', the end-side device 200 determines the motion vector in image frame 2 based on the operation information of the first operation or the second operation.

[0310] Here, when the source-side device 100 displays the most recently captured image frame (e.g., image frame 1) before capturing image frame 2, it moves the page content in the motion area according to the motion vector. After the movement, image frame 2 is displayed. Movement vector. This indicates the distance and direction of movement. In other words, comparing image frame 1 and image frame 2, a certain displayed content or target point in image frame 1 moves according to the movement vector. After the movement, it is displayed in the motion area of ​​image frame 2. In other words, when the source-side device 100 displays image frame 1 and image frame 2 on the display screen, the movement vector when the same content moves from its position in image frame 1 to its position in image frame 2 is this movement vector.

[0311] If the source-side device 100 executes S107, the end-side device 200 can determine the motion vector in image frame 2 based on the operation information of the first operation sent by the source-side device 100. Alternatively, if the end-side device 200 receives the second operation and triggers the source-side device 100 to execute S106, the end-side device 200 can determine the motion vector of image frame 2 based on the operation information of the received second operation.

[0312] In some implementations, the movement vector of the source-side device 100 when scrolling content in a scrollable area is only related to the corresponding operation. Therefore, the end-side device 200 can directly determine the movement vector of the image frame 2 based on the operation information of the first or second operation. For example, the direction of movement of the first operation or the second operation is taken as the direction of movement of image frame 2, and the product of the speed of the first operation or the second operation in the direction of movement and the duration between two adjacent synchronization time points is taken as the length of movement of image frame 2.

[0313] In some implementations, the movement vector and corresponding operation of the source-side device 100 when scrolling content in a scrollable area are related to the sliding parameters of the source-side device 100 itself. Therefore, the end-side device 200 can determine the movement vector of image frame 2 based on the operation information of the first or second operation and the sliding parameters of the source-side device 100 itself. For example, different devices may exhibit different page-sliding effects in response to the same first or second operation. This embodiment of the application takes this into account to accurately calculate the movement vector of image frame 2.

[0314] In some implementations, the movement vector and corresponding operation of the source-side device 100 when scrolling content in a scrollable area, as well as the application currently running in the foreground of the source-side device 100, are considered. Therefore, the endpoint device 200 can determine the movement vector of image frame 2 based on the operation information of the first or second operation and the application running in the foreground of the source-side device 100. For example, when the source device 100 runs different applications, it may present different page-sliding effects in response to the same first or second operation. This embodiment of the application takes this into account to accurately calculate the movement vector of image frame 2.

[0315] As can be seen, S1091'-S1092' above determine the motion region and movement vector of image frame 2 relative to the previous image frame captured by the source device 100 by inference.

[0316] S1093', refer to S1093.

[0317] In other words, in this embodiment of the application, the end-side device 200 can predict the predicted image frame corresponding to the first time point based on one or two image frames that have just entered the display queue.

[0318] refer to Figure 7 , Figure 7 An example is shown of the process by which the end-side device 200 acquires a predicted image frame based on image frame 2.

[0319] like Figure 7 As shown, the end-side device 200 determines the motion region 701 and the motion vector in image frame 2 using the method shown in any of the above embodiments. The end-side device 200 moves the content originally displayed in the motion area 701 according to the movement vector. Moved. After moving, it can be seen that some content originally displayed in the motion area 701, such as content 701a, has been moved out of the motion area, and some areas of the original motion area 701 are now empty, such as... Figure 7 Region 701b in the middle.

[0320] like Figure 7 As shown, the end-side device 200 obtains prediction data based on the content originally displayed in region 701b of image frame 2, and fills it into region 701b to obtain the predicted image frame 5. For example, the clarity of the content displayed in region 701b of the predicted image frame 5 is lower than the clarity of the content originally displayed in region 701b of image frame 2.

[0321] After predicting the first predicted image frame corresponding to the first time point, the edge device 200 can continuously predict subsequent predicted image frames corresponding to the first time point. The method by which the edge device 200 predicts subsequent predicted image frames corresponding to the first time point can also refer to any of the methods in the two embodiments described above. Specifically, the edge device 200 can predict the second predicted image frame corresponding to the first time point based on the latest image frame entering the display queue and the predicted image frame corresponding to the first time point; based on the predicted image frame corresponding to the first time point and the predicted image frame corresponding to the second time point, it can predict the third predicted image frame corresponding to the first time point, and so on. The step of the edge device 200 predicting a new predicted image frame based on two image frames can refer to operations similar to S1091-S1093. Alternatively, the edge device 200 can predict the second predicted image frame corresponding to the first time point based on the predicted image frame corresponding to the first time point; based on the predicted image frame corresponding to the second time point, it can predict the third predicted image frame corresponding to the first time point, and so on. The steps by which the end-side device 200 predicts a new prediction image frame based on an image frame can be referred to as similar operations to S1091'-S1093'.

[0322] refer to Figure 7 Assume that the end-side device 200 obtains a predicted image frame 6 based on image frame 2 and predicted image frame 5. For example, the clarity of the content displayed in region 701b by predicted image frame 6 is lower than the clarity of the content originally displayed in region 701b by predicted image frame 5.

[0323] Since the image frames in the display queue of the edge device 200 change in real time, the edge device 200 will execute S109 multiple times. Therefore, the first time point and predicted image frame determined by the edge device 200 each time S109 is executed can be different. The first time point and predicted image frame determined by the edge device 200 in the most recent execution of S109 will overwrite the results of previous executions of S109, and the previously determined first time point and predicted image frame will become invalid.

[0324] In some implementations, the end-side device 200 can sequentially send the predicted image frames into the prediction image queue, with the predicted image frame corresponding to the first time point being sent into the prediction image queue first. Since S109 is executed multiple times, the image frames in the prediction image queue are updated each time S109 is executed.

[0325] For example, refer to Figure 6 If the end device 200 determines the predicted image frame 5 and the predicted image frame 6 based on image frame c and image frame d (i.e. image frame 1 and image frame 2) during the most recent execution of S109, then the predicted image frame 5 and the predicted image frame 6 will be sent into the predicted image queue in sequence.

[0326] S110, when the synchronization time point arrives, if there are image frames in the display queue, the end device 200 displays the image frames in the display queue; if there are no image frames in the display queue, the end device 200 displays the image frames in the predicted image queue.

[0327] Specifically, the end-side device 200 can continuously generate synchronization signals at this fixed frequency. At the synchronization time point when the synchronization signal arrives, it retrieves an image frame from the current display queue according to the first-in-first-out principle for display. If there is no image frame in the current display queue, it retrieves a predicted image frame from the current prediction image queue according to the first-in-first-out principle for display.

[0328] refer to Figure 6 Assume that image frame c and image frame d are received sequentially in the display queue. Image frame c is provided to the display screen of the end device 200 for display when synchronization signal 3 arrives, and image frame d is provided to the display screen of the end device 200 for display when synchronization signal 4 arrives. After synchronization signal 4, there are no image frames in the display queue. Then, when synchronization signal 5 arrives, the end device 200 retrieves prediction image frame 5 from the prediction image queue for display, and retrieves prediction image frame 6 from the prediction image queue for display when synchronization signal 6 arrives. Similar to the display queue, prediction image frames that have already been provided to the display screen are no longer stored in the prediction image queue, but can be stored in the buffer of the end device 200.

[0329] In this embodiment, the edge device 200 can display image frames from the aforementioned display queue or prediction image queue in full screen, or it can display image frames from the aforementioned display queue or prediction image queue in a partial area of ​​the display screen. In this embodiment, the area on the display screen of the edge device 200 used to display the image frames from the aforementioned display queue or prediction image queue is referred to as the mirror projection area or projection area. It can be seen that the mirror projection area can be the entire area of ​​the display screen or a partial area of ​​the display screen.

[0330] In some embodiments, the size ratio of the mirrored projection area and the display screen in the source device 100 may be the same or different. If they are the same, the end device 200 can display the corresponding image frame in the mirrored projection area proportionally. If they are different, the end device 200 can stretch or scale the image frame according to the size of the mirrored projection area before displaying it to adaptively match the mirrored projection area.

[0331] The position, size, shape, etc. of the mirrored projection area on the display screen of the terminal device 200 can be set by the terminal device 200 by default, or can be set or adjusted by the user. This application embodiment does not limit this.

[0332] If the mirrored projection area only occupies a portion of the display screen of the terminal device 200, then in addition to displaying image frames from the aforementioned display queue or prediction image queue in the mirrored projection area, the terminal device 200 can also display content provided by itself in other areas of the mirrored projection area. The content displayed in other areas of the mirrored projection area depends on the application currently running on the terminal device 200 and the interface currently open, such as interfaces provided by the desktop or other applications, etc., which are not limited here.

[0333] Figures 8A-8D An example is shown of the user interface displayed by the end-side device 200 during screen mirroring.

[0334] like Figures 8A-8D As shown, the mirrored projection area 801 occupies a portion of the display screen of the terminal device 200, while the remaining area of ​​the display screen shows the desktop of the terminal device 200.

[0335] After the source-side device 100 and the end-side device 200 establish a communication connection, assuming that the source-side device 100 intercepts... Figures 5B-5E The four image frames shown, namely image frames 1-4, are sent to the end device 200. Due to communication quality or latency issues, image frames 1-2 are sent to the display queue of the end device 200, while image frames 3-4 are discarded during communication or are discarded due to being outdated.

[0336] Figures 8A-8D The user interface displayed by the end device 200 at several synchronization points is shown.

[0337] like Figure 8A As shown, firstly, the end device 200 displays image frame 1 in the mirror projection area 801.

[0338] like Figure 8B As shown, the end device 200 then displays image frame 2 in the mirror projection area 801.

[0339] like Figure 8C As shown, the end-side device 200 then displays the predicted image frame 5 in the mirror projection area 801.

[0340] like Figure 8D As shown, finally, the end-side device 200 displays the predicted image frame 6 in the mirror projection area 801.

[0341] Among them, predicted image frame 5 and predicted image frame 6 are predicted image frames predicted by the end device 200 in S109.

[0342] Will Figures 8A-8D and Figures 5B-5E In comparison, it is evident that Figure 8A In the mirrored projection area 801, the source device 100 is displayed. Figure 5B Image frame 1 extracted from the image. Figure 8B In the mirrored projection area 801, the source device 100 is displayed. Figure 5C Image frame 2 was captured from the image. Afterwards, due to the source device 100 in... Figure 5E Image frame 3 and Figure 5E Image frame 4 captured from the image was missing a frame, therefore the end device 200... Figure 8A The predicted image frame 5 is displayed in the mirrored projection area 801. Figure 8B The predicted image frame 6 is displayed in the mirrored projection area 801.

[0343] pass Figure 3 The illustrated method flow shows that when all image frames in the display queue of the endpoint device 200 have been displayed, and a new image frame has not yet arrived, the endpoint device 200 can display a predicted image frame. This ensures that the display frame rate of the endpoint device 200 is close to or equal to the projection frame rate of the source device 100, maintaining continuous image display and preventing stuttering. Furthermore, image prediction based on the latest received image frame ensures smoothness and stability of the image, avoiding visual jumps. From the user's perspective, they can see a smooth and continuous image without any stuttering or jumping, resulting in a good projection experience.

[0344] In the above Figure 3 The method flow shown and in other embodiments:

[0345] The image frame that most recently enters the display queue on the end-side device 200 can be referred to as the first image frame, and the second-to-last image frame that recently enters the display queue can be referred to as the second image frame. For example, image frame 2 mentioned in S109 is the first image frame, and image frame 1 is the second image frame.

[0346] Image frames captured and sent to the end device 200 after the source-side device 100 captures the first image frame, but with frame drops, can be referred to as the third image frame. For example, the image frame captured by the source-side device 100... Figures 5D-5E If image frames 3 and 4 are lost due to communication issues, then this is the third image frame.

[0347] The predicted image frame at the first first time point predicted by the end-side device 200 can be referred to as the first predicted image frame, and the predicted image frame at the second first time point can be referred to as the second predicted image frame. For example, predicted image frame 5 can be the first predicted image frame, and predicted image frame 6 can be referred to as the second predicted image frame.

[0348] When the end-side device 100 predicts the predicted image frame corresponding to the first first time point, if the prediction is based on the two most recently submitted image frames to the display queue, then the image frame preceding the most recently submitted image frame can be referred to as the fourth image frame. If the prediction is based on the most recently submitted image frame to the display queue, then the image frame most recently captured by the source-side device 100 before the most recently submitted image frame can also be referred to as the fourth image frame. For example, referring to S109 above, image frame 1 can be the fourth image frame.

[0349] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0350] This application also provides an electronic device, which may include a memory and a processor. The memory may be used to store a computer program; the processor may be used to invoke the computer program in the memory, causing the electronic device to execute the method executed by either the source-side device 100 or the end-side device 200 in any of the above embodiments.

[0351] This application also provides a chip system including at least one processor for implementing the functions involved in the methods executed by either the source-side device 100 or the end-side device 200 in any of the above embodiments.

[0352] In one possible design, the chip system also includes a memory for storing program instructions and data, which may be located within or outside the processor.

[0353] The chip system can consist of chips or include chips and other discrete components.

[0354] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0355] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application embodiment does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application embodiment does not specifically limit the type of memory or the arrangement of the memory and processor.

[0356] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0357] This application also provides a computer program product, the computer program product comprising: a computer program (also referred to as code or instructions), which, when the computer program is run, causes a computer to execute the method executed by either the source-side device 100 or the end-side device 200 in any of the above embodiments.

[0358] This application also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When the computer program is run, it causes the computer to perform the method executed by either the source-side device 100 or the end-side device 200 in any of the above embodiments.

[0359] It should be understood that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by the integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0360] Additionally, this application also provides an apparatus. Specifically, the apparatus may be a component or module, and may include one or more processors and a memory connected together. The memory stores a computer program. When the computer program is executed by one or more processors, the apparatus performs the networking methods described in the above-described method embodiments.

[0361] The apparatus, computer-readable storage medium, computer program product, or chip provided in the embodiments of this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0362] The various embodiments of this application can be combined arbitrarily to achieve different technical effects.

[0363] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0364] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0365] In summary, the above description is merely an embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the disclosure of this application should be included within the scope of protection of this application.

Claims

1. A method for smoothly displaying an image during screen projection, characterized in that, The method is applied to a communication system including a first device and a second device, and the method includes: The first device and the second device establish a communication connection for screen mirroring. The first device captures the content displayed on the screen according to the determined projection frame rate, obtains multiple image frames, and sends the multiple image frames to the second device; The first device synchronizes the screen projection frame rate to the second device; The second device receives the multiple image frames; The second device sends the received multiple image frames into the display queue, determines the local display frequency based on the projection frame rate, retrieves the multiple image frames from the display queue, and displays them at the determined local display frequency. The method further includes: The first device launches the first application and displays the interface of the first application on the screen, and the first device synchronizes the application information of the first application to the second device; The first device receives the user's first operation and synchronizes the operation information of the first operation to the second device; If the speed of the first operation is greater than a first value, the second device predicts the predicted image frame corresponding to the first time point based on the latest image frame entering the display queue, and displays the predicted image frame. The second device makes the prediction based on the application information of the first application, the operation information of the first operation, and the latest image frame entering the display queue.

2. The method according to claim 1, characterized in that, Before the second device displays the predicted image frame, the method further includes: The second device did not receive subsequent image frames from the plurality of image frames, or the second device did not receive subsequent image frames within a first time period.

3. The method according to claim 1, characterized in that, The second device receives the time point of the first image frame among the multiple image frames, and the first device obtains the time point of the first image frame within a first duration.

4. The method according to claim 1, characterized in that, The time point at which the second device receives the second image frame from the plurality of image frames, and the time point at which the first device obtains the second image frame, within a first duration.

5. The method according to claim 1, characterized in that, The predicted image frame is: based on the latest image frame that enters the display queue, the content displayed in the motion area is moved according to the movement vector, and the empty area in the motion area after the movement is filled with the predicted data to obtain the predicted image frame. Wherein, the motion region is the region in the latest image frame entering the display queue where the displayed content has changed in the fourth image frame; the movement vector is the vector that moves the position of the target content in the fourth image frame to the position of the target content in the latest image frame entering the display queue; the empty area in the motion region after the movement is the area in the motion region where no content is displayed after the content displayed in the motion region has been moved; the fourth image frame is the image frame obtained by the first device from the most recent capture of the content displayed on the screen before the latest image frame entering the display queue, or it is the image frame most recently displayed by the second device before displaying the latest image frame entering the display queue; The prediction data is obtained by the second device based on the content displayed in the area vacated in the motion region after the latest image frame entering the display queue.

6. The method according to any one of claims 1-4, characterized in that, The display screen of the second device includes a projection area for displaying the multiple image frames and the predicted image frame, and the projection area occupies part or all of the display screen of the second device.

7. The method according to any one of claims 1-4, characterized in that, The faster the content displayed on the screen of the first device changes, the higher the frame rate at which the first device captures the content displayed on the screen for projection.

8. The method according to any one of claims 1-4, characterized in that, The display frame rate of the second device for displaying the multiple image frames and the predicted image frames is equal to the projection frame rate of the content displayed on the screen captured by the first device.

9. A method for smoothly displaying an image during screen projection, characterized in that, The method is applied to a second device, and the method includes: The second device establishes a communication connection with the first device for screen mirroring. The second device receives multiple image frames sent by the first device, and the multiple image frames are obtained by the first device from the content displayed on the screen according to a determined projection frame rate; The second device sends the received multiple image frames into the display queue, determines the local display frequency based on the projection frame rate, retrieves the multiple image frames from the display queue, and displays them at the determined local display frequency. The method further includes: The second device receives application information of the first application synchronized by the first device; The second device receives the operation information of the first operation synchronized by the first device; If the speed of the first operation is greater than a first value, the second device predicts the predicted image frame corresponding to the first time point based on the latest image frame entering the display queue, and displays the predicted image frame. The second device makes the prediction based on the application information of the first application, the operation information of the first operation, and the latest image frame entering the display queue.

10. The method according to claim 9, characterized in that, Before the second device displays the predicted image frame, the method further includes: The second device did not receive subsequent image frames from the plurality of image frames, or the second device did not receive subsequent image frames within a first time period.

11. The method according to claim 9, characterized in that, The second device receives the time point of the first image frame among the multiple image frames, and the first device obtains the time point of the first image frame within a first duration.

12. The method according to claim 9, characterized in that, The time point at which the second device receives the second image frame from the plurality of image frames, and the time point at which the first device obtains the second image frame, within a first duration.

13. The method according to claim 9, characterized in that, The predicted image frame is: based on the latest image frame that enters the display queue, the content displayed in the motion area is moved according to the movement vector, and the empty area in the motion area after the movement is filled with the predicted data to obtain the predicted image frame. Wherein, the motion region is the region in the latest image frame entering the display queue where the displayed content has changed in the fourth image frame; the movement vector is the vector that moves the position of the target content in the fourth image frame to the position of the target content in the latest image frame entering the display queue; the empty area in the motion region after the movement is the area in the motion region where no content is displayed after the content displayed in the motion region has been moved; the fourth image frame is the image frame obtained by the first device from the most recent capture of the content displayed on the screen before the latest image frame entering the display queue, or it is the image frame most recently displayed by the second device before displaying the latest image frame entering the display queue; The prediction data is obtained by the second device based on the content displayed in the area vacated in the motion region after the latest image frame entering the display queue.

14. The method according to any one of claims 9-12, characterized in that, The display screen of the second device includes a projection area for displaying the multiple image frames and the predicted image frame, and the projection area occupies part or all of the display screen of the second device.

15. The method according to any one of claims 9-12, characterized in that, The display frame rate of the second device for displaying the multiple image frames and the predicted image frames is equal to the projection frame rate of the content displayed on the screen captured by the first device.

16. An electronic device, characterized in that, include: A memory, and one or more processors; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 9-15.

17. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 9-15.

18. A computer program product, characterized in that, The computer program product includes a computer program that, when run, implements the method as described in any one of claims 9-15.

Images