Screen projection method, electronic device, storage medium and program product

Abstract

The embodiment of the application provides a screen projection method, an electronic device, a storage medium and a program product, relates to the technical field of screen projection, and can dynamically adjust the code rate of a screen projection video without affecting the subjective experience of a user as much as possible, reduces power consumption, and improves the user experience. The screen projection method comprises the following steps: acquiring a first application type weight, the first application type weight being an application type weight corresponding to a current foreground application; determining whether to perform code rate adjustment according to a code rate adjustment parameter, the code rate adjustment parameter comprising the first application type weight; if yes, encoding and wirelessly projecting to-be-projected content based on a target code rate, the target code rate being lower than an original code rate of the to-be-projected content; and if no, encoding and wirelessly projecting to-be-projected content based on the original code rate.

Description

Technical Field

[0001] This application relates to the field of screen mirroring technology, specifically to a screen mirroring method, electronic device, storage medium, and program product. Background Technology

[0002] With the increasing interactivity between electronic devices, more and more users are utilizing screen mirroring. For example, when playing videos on a mobile phone, users can use screen mirroring to wirelessly transmit the content from their phone screen to a large-screen TV. However, for mobile phones, screen mirroring requires encoding and transmitting a large amount of video stream data, resulting in high power consumption and potentially causing the phone to overheat during the process. To address this overheating issue, existing technologies monitor the phone's temperature. If the phone gets too hot during mirroring, the bitrate is reduced to lower power consumption. However, this reduces the clarity or smoothness of the mirrored image, thus degrading the user experience. Summary of the Invention

[0003] In view of this, this application provides a screen casting method, electronic device, storage medium, and program product that can dynamically adjust the bitrate of the cast video without affecting the user's subjective experience as much as possible, so as to improve the user experience.

[0004] Firstly, a screen casting method is provided, comprising: obtaining a first application type weight, wherein the first application type weight is the application type weight corresponding to the current foreground application; determining whether to perform bitrate adjustment based on bitrate adjustment parameters, wherein the bitrate adjustment parameters include the first application type weight; if so, encoding and wirelessly casting the content to be cast based on a target bitrate, wherein the target bitrate is lower than the original bitrate of the content to be cast; otherwise, encoding and wirelessly casting the content to be cast based on the original bitrate.

[0005] The current foreground application corresponds to the scenario of the content to be projected. In relatively stable scenarios, where image continuity is good and the probability of application switching is relatively low, the application type corresponding to this type has a higher weight. In more complex scenarios, the application type corresponding to this type has a higher weight because a higher bitrate indicates a more complex scenario, and a higher bitrate means a smaller impact on the user from reducing the bitrate. In other words, the power consumption benefit of adjusting the bitrate is greater, hence the higher weight of this application type. Based on a higher application type weight, it is easier to determine whether to adjust the bitrate. In this scenario, encoding and wireless projection based on the reduced bitrate can reduce power consumption and have a smaller impact on the user's perception. However, for some application types, there may be unstable or low-complexity scenarios. In these scenarios, reducing the bitrate has a greater impact on the user's perception, so it is easier to determine whether not to adjust the bitrate, that is, to encode and wirelessly project the content to be projected based on the original bitrate. In addition, the screen casting method in this application embodiment does not need to wait for the temperature to rise to the threshold before adjusting the bitrate. Instead, it dynamically adjusts the bitrate based on the foreground application type, which can improve the temperature rise of the screen casting source device caused by screen casting, extend the temperature rise time, and reduce the probability of the device malfunctioning due to heat.

[0006] In one possible implementation, the application type weight corresponding to a video or game application is greater than the application type weight corresponding to other applications.

[0007] In one possible implementation, determining whether to perform bitrate adjustment based on the bitrate adjustment parameter includes: if the weight of the first application type is greater than a preset weight, then determining to perform bitrate adjustment; if the weight of the first application type is not greater than the preset weight, then determining not to perform bitrate adjustment.

[0008] In one possible implementation, the bitrate adjustment parameter also includes the frequency with which the current foreground application switches from the background to the foreground within a preset time period. The frequency with which the current foreground application switches from the background to the foreground within the preset time period is continuous, and this switching frequency can reflect the upcoming scenario. Therefore, dynamically adjusting the projection bitrate in conjunction with the switching frequency can further improve the user experience.

[0009] In one possible implementation, determining whether to perform bitrate adjustment based on bitrate adjustment parameters includes: determining a bitrate adjustment index based on the bitrate adjustment parameters; if the necessity of bitrate adjustment represented by the bitrate adjustment index meets the adjustment conditions, then determining to perform bitrate adjustment; if the necessity of bitrate adjustment represented by the bitrate adjustment index does not meet the adjustment conditions, then determining not to perform bitrate adjustment. The necessity of bitrate adjustment represented by the bitrate adjustment index is positively correlated with the weight of the first application type, and the necessity of bitrate adjustment represented by the bitrate adjustment index is negatively correlated with the frequency.

[0010] In one possible implementation, the bitrate adjustment parameter further includes a second application type weight, which is the application type weight corresponding to the target application that is switched to the foreground application during a preset time period. The target application is an application other than the current foreground application. The target application can also reflect the scene of the projected screen to some extent. Therefore, dynamically adjusting the projection bitrate in conjunction with the second application type weight can further improve the user experience.

[0011] In one possible implementation, determining whether to perform bitrate adjustment based on bitrate adjustment parameters includes: determining a bitrate adjustment index based on the bitrate adjustment parameters; if the necessity of bitrate adjustment represented by the bitrate adjustment index meets the adjustment conditions, then determining to perform bitrate adjustment; if the necessity of bitrate adjustment represented by the bitrate adjustment index does not meet the adjustment conditions, then determining not to perform bitrate adjustment. The necessity of bitrate adjustment represented by the bitrate adjustment index is positively correlated with the weight of a first application type, positively correlated with the weight of a second application type, and negatively correlated with the aforementioned frequency.

[0012] In one possible implementation, determining whether to perform bitrate adjustment based on the bitrate adjustment parameter includes: determining the bitrate adjustment index k according to the following formula: k = W app1 2 / (a×f / W app2 If k is greater than the threshold, then bitrate adjustment is determined; if k is not greater than the threshold, then bitrate adjustment is determined not to be performed. app1 The first application type weight is f, where f is the frequency with which the current foreground application switches from the background to the foreground within a preset time period, and W is the weight of the first application type. app2 Here, 'a' represents the weight for the second application type, and 'a' is a constant. Determining the bitrate adjustment index based on this formula allows for more accurate dynamic adjustment of the encoding.

[0013] In one possible implementation, encoding and wirelessly projecting the content to be projected based on a target bitrate includes: determining the target resolution of the content to be projected based on the target bitrate, wherein the target resolution of the content to be projected is lower than the original resolution of the content to be projected; and encoding and wirelessly projecting the content to be projected based on the target resolution. This can minimize the negative impact on user experience while reducing the bitrate.

[0014] Secondly, a screen casting method is provided, including: determining whether to adjust the bitrate based on the frequency of the current foreground application switching from the background to the foreground within a preset time period; if so, encoding and wirelessly casting the content to be cast based on the target bitrate, wherein the target bitrate is lower than the original bitrate of the content to be cast; if not, encoding and wirelessly casting the content to be cast based on the original bitrate.

[0015] Based on the user's historical operation behavior, the probability of the user switching to the foreground application in the coming period can be predicted. If the probability of the user switching to the foreground application is high, it is more likely that bitrate adjustment will not be performed, because if the bitrate of the switched foreground application decreases, it will have a significant impact on the user's subjective experience, which can easily degrade the user experience. If the probability of the user switching to the foreground application is low, it is more likely that bitrate adjustment will be performed, because in stable scenarios, the decrease in bitrate has a smaller impact on the user's perception, and it is less likely to degrade the user experience. Dynamic bitrate adjustment based on the switching frequency of foreground applications can improve the user experience while reducing power consumption. In addition, the screen casting method in this embodiment does not need to wait for the temperature to rise to a threshold before adjusting the bitrate, but rather dynamically adjusts the bitrate based on the switching frequency of foreground applications. This can mitigate the temperature rise of the screen casting source device caused by screen casting, extend the temperature rise time, and reduce the probability of device malfunctions due to overheating.

[0016] In one possible implementation, determining whether to perform bitrate adjustment based on the frequency with which the current foreground application switches from the background to the foreground within a preset time period includes: determining whether to perform bitrate adjustment based on the frequency with which the current foreground application switches from the background to the foreground within the preset time period and a second application type weight, where the second application type weight is the application type weight corresponding to the target application that is switched to the foreground within the preset time period, and the target application is an application other than the current foreground application. Combining the switching frequency of the foreground application and the application type of the target application switched to in order to determine whether to perform bitrate adjustment can further improve the user experience.

[0017] In one possible implementation, the application type weight corresponding to a video or game application is greater than the application type weight corresponding to other applications.

[0018] In one possible implementation, determining whether to perform bitrate adjustment based on the frequency of the current foreground application switching from the background to the foreground within a preset time period and the weight of the second application type includes: determining a bitrate adjustment index based on the frequency of the current foreground application switching from the background to the foreground within a preset time period and the weight of the second application type; if the necessity of bitrate adjustment represented by the bitrate adjustment index meets the adjustment conditions, then bitrate adjustment is determined to be performed; if the necessity of bitrate adjustment represented by the bitrate adjustment index does not meet the adjustment conditions, then bitrate adjustment is determined not to be performed. The necessity of bitrate adjustment represented by the bitrate adjustment index is positively correlated with the weight of the second application type, and the necessity of bitrate adjustment represented by the bitrate adjustment index is negatively correlated with the aforementioned frequency. The necessity of bitrate adjustment represented by the bitrate adjustment index meeting the adjustment conditions may mean that the bitrate adjustment index belongs to a first range, and the necessity of bitrate adjustment represented by the bitrate adjustment index not meeting the adjustment conditions may mean that the bitrate adjustment index belongs to a second range.

[0019] In one possible implementation, encoding and wirelessly projecting the content to be projected based on a target bitrate includes: determining the target resolution of the content to be projected based on the target bitrate, wherein the target resolution of the content to be projected is lower than the original resolution of the content to be projected; and encoding and wirelessly projecting the content to be projected based on the target resolution. This can minimize the negative impact on user experience while reducing the bitrate.

[0020] Thirdly, embodiments of this application provide a screen casting method, including: acquiring content that has already been cast; determining the predicted bitrate of the content to be cast; determining the target bitrate of the content to be cast based on the predicted bitrate and the perceptual encoding distortion curve, wherein the target bitrate of the content to be cast is less than the predicted bitrate; encoding the content to be cast based on the target bitrate and wirelessly casting it.

[0021] The method determines the predicted bitrate of the content to be projected, and then determines the target bitrate after reducing the bitrate based on the predicted bitrate and the perceived coding distortion curve. The content to be projected is then encoded and wirelessly projected based on the target bitrate. Because the target bitrate is determined based on the predicted bitrate and the perceived coding distortion curve, the reduced target bitrate has a lower impact on user perception, while also reducing power consumption. In other words, this embodiment can dynamically adjust the bitrate of the projected video without significantly affecting the user's subjective experience, improving user experience while reducing power consumption. Furthermore, the projecting method in this embodiment does not require waiting for the temperature to rise to a threshold before adjusting the bitrate; instead, it dynamically adjusts the bitrate based on the projected content. This can mitigate the temperature rise of the source device caused by projecting, extend the temperature rise time, and reduce the probability of device malfunctions due to overheating.

[0022] In one possible implementation, determining the predicted bitrate of the content to be projected includes determining the predicted bitrate of the content to be projected based on the visual complexity of the already projected content.

[0023] In one possible implementation, the vertical axis of the perceptual coding distortion curve represents the image distortion rate perceived by the human eye, and the horizontal axis represents the bitrate. The perceptual coding distortion curve has multiple steps, where the width of the steps is positively correlated with the bitrate, and the height of the steps is negatively correlated with the bitrate. Determining the target bitrate of the content to be projected based on the predicted bitrate and the perceptual coding distortion curve includes: determining the step corresponding to the predicted bitrate of the content to be projected in the perceptual coding distortion curve; and determining the bitrate to the left of the predicted bitrate in the corresponding step as the target bitrate of the content to be projected. The width of the steps is positively correlated with the bitrate, meaning that under the same image quality, the higher the temporal and spatial complexity of the image, the stronger the visual masking effect. Therefore, if the predicted bitrate is large, the target bitrate can be set down significantly based on the predicted bitrate, while still ensuring that the human eye does not perceive the distortion caused by the reduced bitrate. Among multiple steps, the height of the step is negatively correlated with the bitrate. That is, under the same image quality, the higher the temporal and spatial complexity of the image, the smaller the impact on human visual perception caused by reducing the bitrate. Dynamically adjusting the bitrate based on the above-mentioned perceptual coding distortion curve can further improve the user experience.

[0024] In one possible implementation, determining the predicted bitrate of the content to be projected based on the image complexity of the already projected content includes: determining the target image complexity of the content to be projected based on the image complexity of the already projected content; and determining the predicted bitrate of the content to be projected based on the target image complexity. Under the premise of the same image quality, there is a positive correlation between image complexity and the required bitrate. After obtaining the image complexity of the content to be projected, the bitrate of the content to be projected can be determined based on the correlation between image complexity and bitrate.

[0025] In one possible implementation, determining the target screen complexity of the content to be projected based on the screen complexity of the already projected content includes: calculating the target screen complexity of the content to be projected using the following formula: c N+1 Let N represent the complexity of the target screen content to be projected, N be the number of frames in the projected content, i be the frame number in the projected content, and the value of i is positively correlated with the time series of the projected content. W i C represents the complexity weighting coefficient of the i-th frame in the multi-frame projected content. i The image complexity of the i-th frame in the multi-frame projected content; W i It is positively correlated with i.

[0026] In one possible implementation, L i QP is the encoding length corresponding to the content projected in the i-th frame. iThese are the encoding parameters corresponding to the content projected in the i-th frame.

[0027] In one possible implementation, determining the predicted bitrate of the content to be projected based on the complexity of the target screen includes: calculating the encoding length corresponding to the content to be projected according to the following formula; L N+1 QP is the encoding length corresponding to the content to be projected. N+1 These are the encoding parameters corresponding to the content to be projected. qp i These are the weighting coefficients of the encoding parameters for the i-th frame in the multi-frame cast content. qp i Positively correlated with i, C N+1 The complexity of the target screen for the content to be projected is determined; the predicted bitrate of the content to be projected is calculated using the following formula: Predicted bitrate = L N+1 × frame rate.

[0028] In one possible implementation, the method further includes: obtaining a first application type weight, where the first application type weight is the application type weight corresponding to the current foreground application; determining whether to perform bitrate adjustment based on bitrate adjustment parameters, where the bitrate adjustment parameters include the first application type weight; if the first application type weight is present, then performing encoding and wireless casting of the content to be cast based on the target bitrate; otherwise, encoding and wireless casting of the content to be cast based on the original bitrate. A higher application type weight makes it easier to determine whether to perform bitrate adjustment. In this scenario, encoding and wireless casting based on a dynamically reduced bitrate can reduce power consumption while having a smaller impact on the user's perception. However, for some application types, there may be unstable or low-complexity scenarios. In these scenarios, reducing the bitrate has a greater impact on the user's perception, making it easier to determine whether not to perform bitrate adjustment, i.e., encoding and wireless casting of the content to be cast based on the original bitrate.

[0029] In one possible implementation, the application type weight corresponding to a video or game application is greater than the application type weight corresponding to other applications.

[0030] In one possible implementation, determining whether to perform bitrate adjustment based on bitrate adjustment parameters includes: determining a bitrate adjustment index based on the bitrate adjustment parameters; if the necessity of bitrate adjustment represented by the bitrate adjustment index meets the adjustment conditions, then bitrate adjustment is determined to be performed; if the necessity of bitrate adjustment represented by the bitrate adjustment index does not meet the adjustment conditions, then bitrate adjustment is determined not to be performed. The necessity of bitrate adjustment represented by the bitrate adjustment index is positively correlated with the weight of a first application type. The weight of the first application type is positively correlated with the power consumption benefit of reducing bitrate in the scenario. By dynamically adjusting the projection bitrate based on application type, the user experience can be improved. The necessity of bitrate adjustment represented by the bitrate adjustment index meeting the adjustment conditions may mean that the bitrate adjustment index belongs to a first range, and the necessity of bitrate adjustment represented by the bitrate adjustment index not meeting the adjustment conditions may mean that the bitrate adjustment index belongs to a second range.

[0031] In one possible implementation, the bitrate adjustment parameter also includes the frequency with which the current foreground application switches from the background to the foreground within a preset time period. Determining whether to adjust the bitrate based on the switching frequency of the foreground application can further improve the user experience.

[0032] In one possible implementation, the bitrate adjustment parameter further includes a second application type weight, which is the application type weight corresponding to the target application that is switched to the foreground application during a preset time period. The target application is an application other than the current foreground application. Determining whether to perform bitrate adjustment by combining the switching frequency of the foreground application and the application type of the target application can further improve the user experience.

[0033] In one possible implementation, determining whether to perform bitrate adjustment based on bitrate adjustment parameters includes: determining a bitrate adjustment index based on the bitrate adjustment parameters; if the necessity of bitrate adjustment represented by the bitrate adjustment index meets the adjustment conditions, then determining to perform bitrate adjustment; if the necessity of bitrate adjustment represented by the bitrate adjustment index does not meet the adjustment conditions, then determining not to perform bitrate adjustment. The necessity of bitrate adjustment represented by the bitrate adjustment index is positively correlated with the weight of a first application type, positively correlated with the weight of a second application type, and negatively correlated with the switching frequency.

[0034] In one possible implementation, determining whether to perform bitrate adjustment based on the bitrate adjustment parameter includes: determining the bitrate adjustment index k according to the following formula: k = W app1 2 / (a×f / W app2 If k is greater than the threshold, then bitrate adjustment is determined; if k is not greater than the threshold, then bitrate adjustment is determined not to be performed. app1 The first application type weight is f, where f is the frequency with which the current foreground application switches from the background to the foreground within a preset time period, and W is the weight of the first application type.app2 Here, 'a' represents the weight for the second application type, and 'a' is a constant. Determining the bitrate adjustment index based on this formula allows for more accurate dynamic adjustment of the bitrate.

[0035] In one possible implementation, encoding and wirelessly projecting the content to be projected based on a target bitrate includes: determining the target resolution of the content to be projected based on its target bitrate, wherein the target resolution is lower than the original resolution of the content; and encoding and wirelessly projecting the content based on the target resolution. By maintaining the quality of individual pixels and determining the target resolution by reducing the resolution, bitrate adjustment is achieved. This method of reducing the bitrate results in minimal degradation of the image quality displayed on the target device.

[0036] Fourthly, an electronic device is provided, comprising: a processor and a memory, the memory being used to store at least one instruction, which, when loaded and executed by the processor, causes the electronic device to perform the method described above.

[0037] Fifthly, a computer-readable storage medium is provided, including a program or instructions, wherein the above-described method is executed when the program or instructions are run on a computer.

[0038] Sixthly, a computer program product is provided, the computer program product containing executable instructions, which, when executed on a computer, cause the computer to perform the above-described method. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 These are schematic diagrams illustrating three screen projection scenarios involved in the embodiments of this application;

[0041] Figure 2 This is a schematic diagram of the mechanism of an electronic device according to an embodiment of this application;

[0042] Figure 3 This is a software structure block diagram of an electronic device in an embodiment of this application;

[0043] Figure 4 This is a flowchart illustrating a screen projection method according to an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of a perceptual coding distortion curve in an embodiment of this application;

[0045] Figure 6 This is a flowchart illustrating another screen mirroring method in an embodiment of this application;

[0046] Figure 7 This diagram illustrates the changes in resolution at different stages under three different comparison scales.

[0047] Figure 8 This is a schematic diagram of a resolution encoding relationship in an embodiment of this application;

[0048] Figure 9 This is a schematic diagram of the temperature rise curves comparing the embodiments of this application with other methods;

[0049] Figure 10 This is a flowchart illustrating another screen mirroring method in an embodiment of this application;

[0050] Figure 11 This is a flowchart illustrating another screen projection method in an embodiment of this application. Detailed Implementation

[0051] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0052] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0053] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0054] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0055] First, the application scenarios involved in the embodiments of this application will be described. The embodiments of this application involve wireless screen projection scenarios, such as... Figure 1 As shown, examples include screen mirroring from a mobile phone to a computer, a TV, or a tablet. This screen mirroring can be a type of screen mirroring. Wireless screen mirroring scenarios involve two types of electronic devices: the source device and the target device.

[0056] Figure 2 A schematic diagram of the structure of the electronic device 100 is shown.

[0057] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, wireless communication module 160, display screen 194, antenna 1, etc.

[0058] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0059] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors. Processor 110 can be used for functions such as audio and video stream encoding and decoding.

[0060] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0061] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0062] The wireless communication function of electronic device 100 can be implemented through antenna 1, wireless communication module 160, modem processor and baseband processor, etc.

[0063] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), and other wireless communication technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 1, frequency-modulates and filters the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, frequency-modulate and amplify them, and then convert them into electromagnetic waves for radiation via antenna 1. The wireless communication module 160 may include, for example, a Bluetooth chip.

[0064] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0065] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0066] In some embodiments, the electronic device 100 may have a touch sensor disposed on the display screen 194, forming a touchscreen, also known as a "touchscreen," composed of the touch sensor and the display screen 194. The touch sensor is used to detect touch operations performed on or near it. The touch sensor can transmit the detected touch operation to an application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor may also be disposed on the surface of the electronic device 100, in a different location than the display screen 194.

[0067] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0068] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0069] Figure 3 This is a software structure block diagram of the electronic device 100 according to an embodiment of this application.

[0070] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0071] The application layer can include a series of application packages.

[0072] like Figure 3 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0073] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0074] like Figure 3 As shown, the application framework layer may include a window manager, content provider, view system, resource manager, etc.

[0075] The window manager is used to manage window programs. The content provider is used to store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0076] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views.

[0077] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0078] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0079] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0080] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0081] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0082] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0083] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0084] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0085] A 2D graphics engine is a graphics engine for 2D drawing.

[0086] The kernel layer is the layer between hardware and software. The kernel layer includes at least the display driver, Bluetooth driver, audio driver, and sensor driver.

[0087] The electronic devices involved in the embodiments of this application may be any product such as smart TVs, mobile phones, tablets, personal computers (PCs), personal digital assistants (PDAs), smartwatches, wearable electronic devices, augmented reality (AR) devices, virtual reality (VR) devices, in-vehicle devices, drone devices, smart cars, smart speakers, robots, smart glasses, etc.

[0088] The aforementioned electronic devices can be either the projection source device or the projection target device in a wireless projection scenario.

[0089] This application provides a screen mirroring method, applied to a screen mirroring source device, such as... Figure 4 As shown, the screen mirroring method includes:

[0090] Step 101: Perform wireless screen mirroring;

[0091] Before wireless screen mirroring, a wireless connection is established between the source device and the target device. During the wireless screen mirroring process, the source device captures its own display screen, which is the screen mirroring content, and encodes the screen mirroring content. Then, it sends the encoded screen mirroring content to the target device via the wireless connection. After receiving the screen mirroring content, the target device decodes and plays the screen mirroring content, thus realizing wireless screen mirroring. At the beginning of the screen mirroring, the source device can encode the screen mirroring content and perform wireless screen mirroring based on the original bitrate.

[0092] Step 102: Obtain the content that has been projected;

[0093] During the screen mirroring process, the source device can obtain the screen mirrored content over a recent period of time, such as the screen mirrored content over the last minute.

[0094] Step 103: Determine the predicted bitrate of the content to be cast based on the complexity of the already cast content;

[0095] The content that has been projected consists of frames of images. The complexity of any frame can be determined. In video, the frames are continuous, and there is a strong correlation between adjacent or close frames. The content of the images is usually similar, so the complexity of the images is also similar and there are usually no sudden changes. Therefore, the bitrate of the content to be projected can be predicted based on the image complexity of the most recently projected content, that is, the predicted bitrate of the content to be projected can be determined.

[0096] Understandably, determining the predicted bitrate of the content to be cast based on the complexity of the already cast content is only one way. In other possible implementations, the predicted bitrate of the content to be cast can also be determined in other ways. For example, the predicted bitrate of the content to be cast can be determined by the actual bitrate of the already cast content. For example, the average actual bitrate of the already cast content over a period of time can be used as the predicted bitrate of the content to be cast.

[0097] Step 104: Determine the target bitrate of the content to be projected based on the predicted bitrate and the perceptual coding distortion curve. The target bitrate of the content to be projected is less than the predicted bitrate.

[0098] The perceptual coding distortion curve is a pre-set curve, or mapping relationship, based on the relationship between the image distortion rate and the image bit rate perceived by the human eye. For example, ... Figure 5 As shown, the relationship between the statistical coding distortion curve and the perceptual coding distortion curve is illustrated. The statistical coding distortion curve refers to the theoretical correspondence between image distortion rate and bitrate. From a computer perspective, the correlation between image distortion rate and bitrate is a continuous curve. The perceptual coding distortion curve refers to the correspondence between image distortion rate and bitrate perceived by the human eye. Due to the visual masking effect, the correlation between the perceived image distortion rate and bitrate is reflected in the human eye. That is, the human eye only perceives the distortion rate after the bitrate distortion accumulates to a certain threshold. In other words, the perceptual coding distortion curve has multiple steps, and the change in image bitrate at each step does not correspond to a change in image distortion rate. Only changes in bitrate between different steps will cause a sudden change in image distortion rate. In step 103, the position of the predicted bitrate of the content to be projected in the perceptual coding distortion curve is determined. Assume the predicted bitrate of the content to be projected is... Figure 5In the diagram, A is located at the bottom step of the perceptual coding distortion curve. Since all bitrates on this step correspond to the same image distortion rate perceived by the human eye, the bitrate to the left of A on this step can be set as the target bitrate of the content to be projected. For example, A' can be set as the target bitrate of the content to be projected. The smaller the target bitrate A', the better the power consumption reduction effect. However, the target bitrate A' cannot be equal to or less than the bitrate s1 corresponding to the rising edge on the left side of this step. This is because, on the same platform, the image distortion rate perceived by the human eye remains unchanged, meaning the user will not perceive the distortion caused by the reduction in bitrate. Therefore, s1 < A' < A. The closer A' is to the bitrate s1 corresponding to the rising edge on the left side of this step, the better the bitrate reduction effect. Assume the predicted bitrate of the content to be projected is... Figure 5 If the bitrate corresponding to the left rising edge of the step containing B is s2, then the bitrate B' between s2 and B can be set as the target bitrate of the content to be projected.

[0099] Step 105: Encode the content to be projected based on the target bitrate and project wirelessly.

[0100] The target bitrate is lower than the predicted bitrate, which means that the bitrate of the content to be projected is reduced. As a result, the image bitrate transmitted wirelessly is lower during the wireless projection process, which means that the amount of data transmitted is lower, thereby reducing power consumption. At the same time, since the reduction of the target bitrate is determined based on the perceptual coding distortion curve, the reduction of the bitrate has little impact on human visual perception.

[0101] After step 105, step 102 can be executed again. That is, steps 102 to 105 are executed periodically during the wireless screen mirroring process. To illustrate with a specific scenario, in a wireless screen mirroring process, frames 1 to N are encoded and wirelessly mirrored at a default bitrate. Then, in step 103, the predicted bitrate of the (N+1)th frame to be mirrored is determined based on the complexity of the mirrored content in frames 1 to N. In step 104, the target bitrate of the (N+1)th frame is determined based on the predicted bitrate. Finally, in step 105, the (N+1)th frame is encoded and wirelessly mirrored based on the target bitrate. Then, based on the complexity of the content already projected in frames 2 to N+1, the predicted bitrate of the N+2 frame to be projected is determined, and the target bitrate of the N+2 frame is also determined. The N+2 frame is then encoded and wirelessly projected according to the target bitrate. This process is repeated, with the predicted bitrate of the N+3 frame to be projected determined based on the complexity of the content already projected in frames 3 to N+3, and the target bitrate of the N+3 frame also determined. The N+3 frame is then encoded and wirelessly projected according to the target bitrate. In this way, the bitrate can be dynamically adjusted based on the content being projected during wireless projection.

[0102] The screen casting method in this embodiment obtains the predicted bitrate of the content to be cast based on the already cast content, and determines the target bitrate after reducing the bitrate based on the predicted bitrate and the perceived coding distortion curve. The content to be cast is then encoded and wirelessly cast based on the target bitrate. Because the target bitrate is determined based on the predicted bitrate and the perceived coding distortion curve, the reduced target bitrate has a lower impact on user perception, while also reducing power consumption. In other words, this embodiment can dynamically adjust the bitrate of the cast video without significantly affecting the user's subjective experience, improving user experience while reducing power consumption. Furthermore, the screen casting method in this embodiment does not require waiting for the temperature to rise to a threshold before adjusting the bitrate; instead, it dynamically adjusts the bitrate based on the already cast content. This can mitigate the temperature rise of the source device caused by screen casting, extend the temperature rise time, and reduce the probability of device malfunctions due to overheating.

[0103] In one possible implementation, such as Figure 5 As shown, the vertical axis of the perceptual coding distortion curve represents the image distortion rate perceived by the human eye, and the horizontal axis represents the bitrate. The perceptual coding distortion curve has multiple steps. Step 104, determining the target bitrate of the content to be projected based on the predicted bitrate and the perceptual coding distortion curve, includes: determining the step corresponding to the predicted bitrate of the content to be projected in the perceptual coding distortion curve; and determining the bitrate to the left of the predicted bitrate in the corresponding step as the target bitrate of the content to be projected. The bitrate to the left of the predicted bitrate in the corresponding step does not include the rising edge to the left of the step. The width of each step is positively correlated with the bitrate. That is, under the same image quality, the higher the temporal and spatial complexity of the image, the stronger the visual masking effect. Therefore, if the predicted bitrate is large, the target bitrate can be set down significantly based on the predicted bitrate, while still ensuring that the human eye does not perceive the distortion caused by the reduced bitrate. Among multiple steps, the height of the step is negatively correlated with the bitrate. This means that, for the same image quality, the higher the temporal and spatial complexity of the image, the smaller the impact on human visual perception caused by reducing the bitrate. For example, if the projected content is video, and video content is highly complex, even with a significant reduction in bitrate, users are unlikely to perceive image distortion. However, if the projected content is a chat interface from an instant messaging application, which is less complex, the image distortion caused by reducing the bitrate is easily perceived by the user. Therefore, dynamically adjusting the bitrate based on the aforementioned perceived encoding distortion curve can further improve the user experience.

[0104] The following example illustrates how to determine the target bitrate of the content to be projected using the bitrate located to the left of the predicted bitrate in the corresponding step. For instance, a positive number less than 1 is set as the target bitrate adjustment coefficient, such as 0.8. When determining the target bitrate of the content to be projected, the predicted bitrate of the content to be projected is directly multiplied by the coefficient 0.8 to obtain the target bitrate.

[0105] In one possible implementation, step 103, determining the predicted bitrate of the content to be projected based on the image complexity of the already projected content, includes steps 1031 and 1032. Step 1031: Determine the target image complexity of the content to be projected based on the image complexity of the already projected content. For example, in the content projected in the most recent minute, one frame per second is acquired, resulting in 60 frames. The image complexity of the content to be projected is predicted based on the image complexity of these 60 frames. Alternatively, in the content projected in the most recent second, each frame is acquired based on the display frame rate. Assuming a frame rate of 60Hz, the most recent 60 consecutive projected frames are acquired, and the image complexity of the content to be projected is predicted based on the image complexity of these 60 frames. Step 1032: Determine the predicted bitrate of the content to be projected based on the target image complexity. Under the premise of the same image quality, there is a positive correlation between the image complexity and the required bitrate. After obtaining the screen complexity of the content to be projected, the bitrate of the content to be projected can be determined based on the correlation between screen complexity and bitrate. This bitrate is referred to as the predicted bitrate in this embodiment of the application.

[0106] In one possible implementation, step 1031 above, determining the target screen complexity of the content to be projected based on the screen complexity of the already projected content, includes: calculating the target screen complexity of the content to be projected using the following formula: C N+1 Let N represent the complexity of the target screen content to be projected, N be the number of frames in the projected content, i be the frame number in the projected content, and the value of i is positively correlated with the time series of the projected content. W i C represents the complexity weighting coefficient of the i-th frame in the multi-frame projected content. i The image complexity of the i-th frame in the multi-frame projected content; W i It is positively correlated with i.

[0107] Specifically, multiple frames of already cast content can be selected from the most recently cast content, and the target screen complexity of the content to be cast is determined based on the screen complexity of the selected multiple frames of already cast content. In one embodiment, for example, all 60 frames of already cast content in the most recent second can be used as the multiple frames of already cast content in step 1031, N=60, and the value of i is 1, 2, 3, ..., 60. The value of i is positively correlated with the time series of the multiple frames of already cast content, that is, the larger the value of i, the later the time in the corresponding 60 frames. N+1 and C N It can correspond to two adjacent frames. The closer the frame is to the content to be projected, the higher its correlation with the content. Therefore, the complexity of the content to be projected can be determined based on multiple frames of projected content adjacent to it. Furthermore, for this reason, in the calculation formula, a larger i corresponds to a larger complexity weighting coefficient, thus improving the accuracy of predicting the complexity of the content to be projected. In another embodiment, for example, one frame can be selected every second from the 360 ​​projected frames in the most recent minute, and 60 of these frames can be used as the multiple frames of projected content in step 1031.

[0108] In one possible implementation, L i QP is the encoding length corresponding to the content projected in the i-th frame. i QP represents the encoding parameters corresponding to the content projected in the i-th frame. i The value is usually a natural number between 0 and 51. The larger the value, the lower the image quality. The value may vary between different frames, but QP... i The parameters are known.

[0109] In addition to determining the complexity of the projected content using the formula mentioned above, other methods can also be used. For example, the complexity can be determined by the color distribution of the projected content, which may be related to image parameters such as the variance or standard deviation of RGB pixels. In the embodiments of this application, all are based on C... i The formula for determining the complexity of the projected content will be used as an example for explanation.

[0110] In one possible implementation, step 1032 above, determining the predicted bitrate of the content to be projected based on the complexity of the target screen, includes: calculating the encoding length corresponding to the content to be projected according to the following formula; L N+1 QP is the encoding length corresponding to the content to be projected. N+1 These are the encoding parameters corresponding to the content to be projected. qp iThese are the weighting coefficients of the encoding parameters for the i-th frame in the multi-frame projected content. qp i Positively correlated with i, C N+1 The target screen complexity of the content to be projected is determined; the predicted bitrate of the content to be projected is calculated using the following formula: Predicted bitrate = L N+1 × Frame rate, which can be the frequency at which screen data of the cast content is captured on the source device.

[0111] In one possible implementation, such as Figure 6 As shown, the above method further includes: Step 106, obtaining the first application type weight, where the first application type weight is the application type weight corresponding to the current foreground application; Step 107, determining whether to perform bitrate adjustment based on the bitrate adjustment parameters, where the bitrate adjustment parameters include the first application type weight. If yes, then execute Step 105, the process of encoding and wirelessly casting the content to be cast based on the target bitrate; if no, then execute Step 108, the process of encoding and wirelessly casting the content to be cast based on the original bitrate. After Step 105, Step 106 can be executed again to determine whether to perform bitrate adjustment in the next cycle, thus achieving periodic dynamic bitrate adjustment. After Step 108, Step 106 can be executed again to determine whether to perform bitrate adjustment in the next cycle, thus achieving periodic dynamic bitrate adjustment.

[0112] Specifically, the decision to reduce the casting bitrate can be determined based on the current foreground application type. The first application type weight indicates whether the current foreground application has value for bitrate adjustment; a higher weight indicates a greater likelihood of bitrate adjustment in step 107. The current foreground application corresponds to the scenario of the content to be cast. In a relatively stable scenario, i.e., with good image continuity and a relatively low probability of application switching, the application type weight is higher. In a more complex scenario, the first application type weight is higher, because a higher bitrate indicates a more complex scenario. Figure 5 As shown, the higher the bitrate, the smaller the impact of reducing the bitrate on the user. In other words, the power consumption benefit of bitrate adjustment is greater. Therefore, the first application type has a higher weight in this case. Based on the higher weight of the first application type, it is easier to determine whether to adjust the bitrate in step 107. In this scenario, encoding and wireless projection based on the dynamically reduced bitrate can reduce power consumption while having a smaller impact on the user's perception. However, for some application types, there may be unstable or low-complexity scenarios. In these scenarios, reducing the bitrate has a greater impact on the user's perception. Therefore, it is easier to determine whether not to adjust the bitrate in step 107, that is, to encode and wirelessly project the content to be projected based on the original bitrate in step 108.

[0113] It should be noted that, in Figure 6 In the process shown, if it is determined in step 107, then steps 102 to 105 are executed, that is, the target bitrate is determined after it is determined to perform bitrate adjustment. However, this application embodiment does not limit the order between steps 102 to 104 and step 107. That is, in other possible implementations, the target bitrate can be determined before determining whether to perform bitrate adjustment or during the process of determining bitrate adjustment. Whether to perform encoding and wireless screen projection based on the target bitrate needs to be triggered according to the result determined in step 107.

[0114] In one possible implementation, the bitrate is resolution-dependent and also related to the quality of each pixel. The bitrate is approximately equal to the resolution multiplied by the quality of each pixel. Therefore, bitrate adjustment can be achieved by keeping the resolution constant while reducing the amount of data per pixel, i.e., reducing the quality of each pixel. In other words, step 105 includes determining the target quality of each pixel corresponding to the content to be projected based on the target bitrate of the content. The target quality of each pixel is less than the original quality of the corresponding pixel. While keeping the resolution constant, the content to be projected is encoded and wirelessly projected based on the target quality of each pixel and the unchanged resolution. Understandably, in other possible approaches, the quality of each pixel and the resolution can also be reduced simultaneously, and the content to be projected is encoded and wirelessly projected based on the reduced quality of each pixel and the reduced resolution.

[0115] In one possible implementation, step 105, encoding the content to be projected and wirelessly projecting it based on the target bitrate, includes: determining the target resolution of the content to be projected based on the target bitrate of the content to be projected, wherein the target resolution of the content to be projected is lower than the original resolution of the content to be projected; and encoding and wirelessly projecting the content to be projected based on the target resolution.

[0116] Specifically, in most screen mirroring scenarios, the source device captures a high-resolution image, while the target device downsamples the received data before displaying it, resulting in a lower resolution display on the target device. For example, ... Figure 7 As shown, Figure 7The size of the rectangle represents the resolution; a larger rectangle indicates a higher resolution. In Comparative Example 1, the image captured by the projection source device has a high resolution. Encoding, transmission, and decoding are performed without changing the resolution, but the resolution displayed on the projection target device is relatively low. In Comparative Example 2, the resolution of encoding, transmission, and decoding is lower than the resolution of the captured image, but the final resolution displayed on the projection target device is even lower. In Comparative Example 3, the resolution of encoding, transmission, and decoding is the same as the resolution displayed on the projection target device. In this method, the power consumption during encoding, transmission, and decoding is the lowest, but the display effect is the same as in the other two comparative examples. Therefore, in this embodiment, by maintaining the quality of a single pixel and determining the target resolution by reducing the resolution, the bitrate is adjusted. This method of reducing the bitrate has a smaller impact on the image quality displayed on the projection target device. Due to encoding limitations, the encoding ratio and resolution ratio have a corresponding relationship under different encoding parameters (QP), as shown in Table 1 and... Figure 8 As shown.

[0117]

[0118] Table 1 shows the correspondence between resolution and bitrate. Figure 8 This is a diagram illustrating the relationship between resolution and bitrate corresponding to Table 1, where both bitrate and resolution are expressed as percentages. Table 1 and Figure 8 In addition to illustrating the correspondence between resolution ratio and bitrate ratio, the diagram also illustrates the correspondence between two reference ratios and bitrate ratios. The single-sided ratio refers to the pixel single-sided ratio. The correspondence between pixel area ratio and bitrate ratio can be equivalently understood as the relationship between ideal resolution and bitrate. However, due to encoding limitations, the actual correspondence between resolution ratio and bitrate is related to the encoding parameter QP. The relationship between resolution ratio and bitrate ratio under different encoding parameters QP can be pre-stored in the projection source device. In the process of determining the target resolution of the content to be projected based on its target bitrate, the ratio of the reduced target bitrate to the original bitrate can be determined, i.e., the reduced bitrate ratio. Based on the resolution ratio and bitrate ratio relationship corresponding to the current encoding parameter QP, the resolution ratio corresponding to the reduced bitrate ratio can be determined. Based on this resolution ratio, the corresponding target resolution can be determined, and then the content to be projected can be encoded and wirelessly projected based on this target resolution.

[0119] As shown in Table 2 and Figure 9 As shown.

[0120] Table 2

[0121]

[0122] Table 2 shows the temperature test data of electronic devices under three scenarios. Figure 9 Table 2 illustrates the temperature rise curves of electronic devices in three different scenarios. It can be seen that the temperature rises faster in the wireless projection scenario than in the local video playback scenario. The wireless projection method of this application embodiment has a slower temperature rise than the wireless projection method without changing the bitrate. In other words, this application embodiment can improve the temperature rise of the projection source device caused by projection and prolong the temperature rise time.

[0123] like Figure 10 As shown in the embodiments of this application, a screen mirroring method is also provided, including:

[0124] Step 201: Obtain the weight of the first application type, which is the weight of the application type corresponding to the current foreground application;

[0125] Step 202: Determine whether to perform bitrate adjustment based on the bitrate adjustment parameters, including the weight of the first application type. If yes, proceed to step 203: Encode and wirelessly project the content to be projected based on the target bitrate, which is lower than the original bitrate of the content to be projected. If no, proceed to step 204: Encode and wirelessly project the content to be projected based on the original bitrate.

[0126] Specifically, the execution subject of this screen projection method can be the screen projection source device. Step 201 can be the same as step 106 in the above embodiment, step 202 can be the same as step 107, step 204 can be the same as step 108, and step 203 can be similar to step 105. The difference is that step 203 does not limit the method of determining the target bitrate. The target bitrate determination method in the above embodiment can be applied, or other possible target bitrate determination methods can be applied, as long as the target bitrate is lower than the original bitrate.

[0127] In the screen casting method of this application embodiment, the current foreground application corresponds to the scenario of the content to be cast. If it is a relatively stable scenario, i.e., the image continuity is good and the probability of application switching is relatively low, then the application type weight corresponding to this application is relatively high. If it is a scenario with higher complexity, then the application type weight corresponding to this application is relatively high, because a higher bitrate indicates a more complex scenario, and a higher bitrate means that reducing the bitrate has less impact on the user. In other words, the power consumption benefit of adjusting the bitrate is greater, so the application type weight corresponding to this application is relatively high. Based on a higher application type weight, it is easier to determine whether to adjust the bitrate. In this scenario, encoding and wireless casting based on the reduced bitrate can reduce power consumption on the one hand, and have less impact on the user's perception on the other hand. However, for some application types, there may be unstable or low-complexity scenarios. In these scenarios, reducing the bitrate has a greater impact on the user's perception. Therefore, it is easier to determine whether not to adjust the bitrate, i.e., to encode and wirelessly cast the content to be cast based on the original bitrate. In addition, the screen casting method in this application embodiment does not need to wait for the temperature to rise to the threshold before adjusting the bitrate. Instead, it dynamically adjusts the bitrate based on the foreground application type, which can improve the temperature rise of the screen casting source device caused by screen casting, extend the temperature rise time, and reduce the probability of the device malfunctioning due to heat.

[0128] In one possible implementation, such as Figure 6 and Figure 10 As shown, step 107 or step 202, determining whether to perform bitrate adjustment based on the bitrate adjustment parameter, includes: if the weight of the first application type is greater than the preset weight, then bitrate adjustment is determined to be performed; if the weight of the first application type is not greater than the preset weight, then bitrate adjustment is determined not to be performed. In other words, in step 107 or 202, the determination of whether to perform bitrate adjustment can be directly based on the weight of the first application type.

[0129] In one possible implementation, such as Figure 6 and Figure 10 As shown, the application type weight corresponding to video or game applications is greater than the application type weight corresponding to other applications. Step 107 or step 202, determining whether to perform bitrate adjustment based on the bitrate adjustment parameters, includes: determining a bitrate adjustment index based on the first application type weight; if the necessity of bitrate adjustment represented by the bitrate adjustment index meets the adjustment conditions, then bitrate adjustment is determined to be performed; if the necessity of bitrate adjustment represented by the bitrate adjustment index does not meet the adjustment conditions, then bitrate adjustment is determined not to be performed. The necessity of bitrate adjustment represented by the bitrate adjustment index is positively correlated with the first application type weight. The first application type weight is positively correlated with the power consumption benefit of reducing bitrate in the scenario. By dynamically adjusting the projection bitrate based on application type, the user experience can be improved.

[0130] Specifically, the bitrate adjustment index is correlated with the weight of the first application type. For example, they can be positively correlated. In this case, if the bitrate adjustment index is greater than the adjustment threshold, meaning the necessity of bitrate adjustment represented by the bitrate adjustment index meets the adjustment condition, then bitrate adjustment is determined to be performed. If the bitrate adjustment index is not greater than the adjustment threshold, meaning the necessity of bitrate adjustment represented by the bitrate adjustment index does not meet the adjustment condition, then bitrate adjustment is determined not to be performed. The bitrate adjustment index and the weight of the first application type can be negatively correlated. In this case, if the necessity of bitrate adjustment represented by the bitrate adjustment index is not greater than the adjustment threshold, meaning the necessity of bitrate adjustment represented by the bitrate adjustment index meets the adjustment condition, then bitrate adjustment is determined to be performed. If the bitrate adjustment index is greater than the adjustment threshold, meaning the necessity of bitrate adjustment represented by the bitrate adjustment index does not meet the adjustment condition, then bitrate adjustment is determined not to be performed. For video and game applications, which typically have high frame rates and are prone to displaying highly complex images, with frame rates usually stable at 30, 60, or 120fps and significant correlation between images, the raw power consumption of these applications when used as the foreground for screen mirroring is relatively high. Therefore, the power consumption benefit of reducing the bitrate is significant. Thus, the application type weight for video or game applications can be set to 1, meaning they have the highest priority in determining whether to adjust the bitrate. For other applications besides video and games, since the probability of displaying highly complex images is low, a lower application type weight can be set. Other applications can be further categorized. For example, music apps dynamically display lyrics while playing songs, with a frame rate typically stable at 30 or 60fps. The image complexity is relatively low compared to games or videos, so the power consumption benefit of reducing the bitrate is moderate and relatively stable. Therefore, the app type weight for music apps can be set to 0.5, meaning they have a medium priority in determining whether to adjust the bitrate. For other apps besides music (i.e., video apps, game apps, and apps other than music apps), such as instant messaging apps or memo apps, take instant messaging apps as an example. When the window is static, its frame rate is usually less than 5fps, resulting in a low casting bitrate. However, when sliding the window or refreshing other functions with image or video interfaces, the frame rate may quickly rise to 60 or 120fps. The response speed for bitrate reduction is not easily matched to the application's frame rate and bitrate change speed, which can easily affect the user's subjective experience. Furthermore, the overall bitrate of these apps is relatively low, so the power consumption benefit of reducing the bitrate is not significant. Therefore, the lowest app type weight should be set, for example, the app type weight for other apps besides music apps can be set to 0. The weight of the first application type is positively correlated with the power consumption benefit of reducing the bitrate in the scenario. By dynamically adjusting the screen projection bitrate based on the application type, the user experience can be improved.

[0131] In one possible implementation, such as Figure 6 and Figure 10As shown, the bitrate adjustment parameters also include the frequency of the current foreground application switching from the background to the foreground within a preset time period. In other words, in step 107 or 202, the bitrate adjustment can be determined based on the weight of the first application type and the frequency of the current foreground application switching from the background to the foreground within a preset time period.

[0132] Specifically, the preset time period is a recent period, such as the last minute. The frequency with which the current foreground application switches from the background to the foreground within the preset time period can characterize the user's historical operational behavior. User operations are usually continuous and correlated. For example, some users habitually switch to instant messaging applications to reply to messages while watching videos, while others rarely switch applications while watching videos. Therefore, based on the user's historical operational behavior, the probability of the user switching to the foreground application in the next period can be predicted. If the probability of the user switching to the foreground application is high, it is more likely that bitrate adjustment will not be performed, because if the bitrate of the foreground application is reduced after switching, it will have a significant impact on the user's subjective experience and may easily degrade the user experience. If the probability of the user switching to the foreground application is low, it is more likely that bitrate adjustment will be performed, because in a stable scenario, the impact of bitrate reduction on the user's perception is small, and bitrate reduction is less likely to degrade the user experience. For example, if a foreground application switches within the last minute, step 204 is executed for the next 10 minutes without reducing the bitrate. Then, step 202 is executed again to determine whether to adjust the bitrate in the next cycle, thus achieving periodic dynamic bitrate adjustment. If no foreground application switches within the last minute, step 203 is executed for the next 10 minutes, reducing the bitrate and casting the screen. Then, step 202 is executed again to determine whether to adjust the bitrate in the next cycle, thus achieving periodic dynamic bitrate adjustment. Determining whether to adjust the bitrate based on the frequency of foreground application switching can further improve the user experience.

[0133] In one possible implementation, such as Figure 6 and Figure 10 As shown, the bitrate adjustment parameters also include a second application type weight, which is the application type weight of the target application that is switched to the foreground application during a preset time period. The target application is an application other than the current foreground application.

[0134] Specifically, if a user frequently switches to an instant messaging app to reply to messages while watching videos recently, it indicates a high probability that the user will switch back to an instant messaging app to reply to messages in the near future. However, due to the different app types of instant messaging and video apps, the impact of reducing bitrate on user experience differs. Therefore, the necessity of bitrate adjustment is not only related to the switching frequency but also to the app type weight corresponding to the target app. For example, if the user has recently switched to a video app, its app type weight is higher, so bitrate adjustment is more likely; conversely, if the user has recently switched to an instant messaging app, its app type weight is lower, so bitrate adjustment is less likely. Combining the switching frequency of the foreground app and the app type of the target app to determine whether to adjust bitrate can further improve the user experience.

[0135] In one possible implementation, such as Figure 6 and Figure 10 As shown, step 107 or 202, determining whether to perform bitrate adjustment based on bitrate adjustment parameters, includes: determining a bitrate adjustment index based on the weight of the first application type and the frequency of the current foreground application switching from the background to the foreground within a preset time period; if the necessity of bitrate adjustment represented by the bitrate adjustment index meets the adjustment conditions, then it is determined to perform bitrate adjustment; if the necessity of bitrate adjustment represented by the bitrate adjustment index does not meet the adjustment conditions, then it is determined not to perform bitrate adjustment. The necessity of bitrate adjustment represented by the bitrate adjustment index is positively correlated with the weight of the first application type, and the necessity of bitrate adjustment represented by the bitrate adjustment index is negatively correlated with the aforementioned frequency.

[0136] In one possible implementation, such as Figure 6 and Figure 10 As shown, step 107 or 202, determining whether to perform bitrate adjustment based on bitrate adjustment parameters, includes: determining a bitrate adjustment index based on the first application type weight and the second application type weight; if the necessity of bitrate adjustment represented by the bitrate adjustment index meets the adjustment conditions, then it is determined to perform bitrate adjustment; if the necessity of bitrate adjustment represented by the bitrate adjustment index does not meet the adjustment conditions, then it is determined not to perform bitrate adjustment. The necessity of bitrate adjustment represented by the bitrate adjustment index is positively correlated with the first application type weight, and the necessity of bitrate adjustment represented by the bitrate adjustment index is positively correlated with the second application type weight.

[0137] In one possible implementation, such as Figure 6 and Figure 10As shown, step 107 or 202, determining whether to perform bitrate adjustment based on bitrate adjustment parameters, includes: determining a bitrate adjustment index based on the weight of the second application type and the frequency of the current foreground application switching from the background to the foreground within a preset time period; if the necessity of bitrate adjustment represented by the bitrate adjustment index meets the adjustment conditions, then bitrate adjustment is determined to be performed; if the necessity of bitrate adjustment represented by the bitrate adjustment index does not meet the adjustment conditions, then bitrate adjustment is determined not to be performed. The necessity of bitrate adjustment represented by the bitrate adjustment index is positively correlated with the weight of the second application type, and the necessity of bitrate adjustment represented by the bitrate adjustment index is negatively correlated with the frequency.

[0138] In one possible implementation, such as Figure 6 and Figure 10 As shown, step 107 or 202, determining whether to perform bitrate adjustment based on bitrate adjustment parameters, includes: determining a bitrate adjustment index based on the weight of the first application type, the weight of the second application type, and the frequency at which the current foreground application switches from the background to the foreground within a preset time period. If the necessity of bitrate adjustment represented by the bitrate adjustment index meets the adjustment conditions, then it is determined to perform bitrate adjustment; if the necessity of bitrate adjustment represented by the bitrate adjustment index does not meet the adjustment conditions, then it is determined not to perform bitrate adjustment. The necessity of bitrate adjustment represented by the bitrate adjustment index is positively correlated with the weight of the first application type, positively correlated with the weight of the second application type, and negatively correlated with the frequency.

[0139] Specifically, both the current foreground application and the recently switched target application could potentially be the application corresponding to the upcoming screen mirroring content. Both reflect the upcoming screen mirroring scenario. Therefore, the necessity of bitrate adjustment, as represented by the bitrate adjustment metric, can be positively correlated with the weights of the first and second application types. In other words, the higher the weight of the application type that was previously switched to the foreground, the greater the probability of a higher bitrate in the upcoming screen mirroring scenario, thus favoring bitrate adjustment. Conversely, a higher switching frequency indicates lower stability in the upcoming screen mirroring scenario, thus favoring not adjusting bitrate. Combining the switching frequency of the foreground application, the application type of the target application, and the application type of the current foreground application to determine whether to adjust bitrate can further improve the user experience.

[0140] In one possible implementation, determining whether to perform bitrate adjustment based on the bitrate adjustment parameter includes: determining the bitrate adjustment index k according to the following formula: k = W app1 2 / (a×f / W app2 If k is greater than the threshold, then bitrate adjustment is determined; if k is not greater than the threshold, then bitrate adjustment is determined not to be performed. app1The first application type weight is f, where f is the frequency with which the current foreground application switches from the background to the foreground within a preset time period, and W is the weight of the first application type. app2 Here, 'a' represents the weight for the second application type, and 'a' is a constant. Determining the bitrate adjustment index based on this formula allows for more accurate dynamic adjustment of the encoding.

[0141] Specifically, a×f / W app2 This can be used to represent the estimated probability of switching to an application type that should not undergo bitrate adjustment within a certain period of time. Assuming that in the first scenario, the current foreground application is a game application, its corresponding application type weight W... app1 =1, the user switches to the music app once in the last minute. The switching frequency of once per minute is set to, for example, f=1. The music app is the target app, and its corresponding app type weight W. app2 =0.5; Assuming the second scenario is similar to the first, the only difference is that the user switches to the music application 5 times in the last minute. If this switching frequency of 5 times per minute is set to f=5, then compared to the first scenario, the switching frequency in the second scenario is higher. Therefore, the bitrate adjustment index k is smaller, making it less likely to exceed the threshold, i.e., less likely to require bitrate adjustment; Assuming the second scenario is similar to the first, the only difference is that the current foreground application is an instant messaging application, and its corresponding application type weight W... app1 =0, the bitrate adjustment index k=0, the threshold is positive, k must be less than the threshold, so no bitrate adjustment is performed; assuming the second scenario is similar to the first scenario, the only difference is that the target application is an instant messaging application, and its corresponding application type weight W app2 =0, bitrate adjustment index k=0, the threshold is positive, k must be less than the threshold, therefore no bitrate adjustment is performed. To ensure normal calculation, if there is no switching within the preset time period, f and W can be set. app2 The default value is non-zero. Determining the bitrate adjustment index based on this formula allows for more accurate dynamic adjustment of the bitrate.

[0142] In one possible implementation, step 203, encoding the content to be projected and wirelessly projecting it based on the target bitrate, includes: determining the target resolution of the content to be projected based on the target bitrate, wherein the target resolution of the content to be projected is lower than the original resolution of the content to be projected; and encoding and wirelessly projecting the content to be projected based on the target resolution. This approach can minimize the negative impact on user experience while reducing the bitrate. The specific process and principle are the same as those in step 105 of the above embodiments, and will not be repeated here.

[0143] like Figure 11 As shown in the embodiments of this application, a screen mirroring method is also provided, including:

[0144] Step 301: Determine whether to adjust the bitrate based on the frequency of the current foreground application switching from the background to the foreground within a preset time period. If yes, proceed to step 302: Encode and wirelessly project the content to be projected based on the target bitrate, which is lower than the original bitrate of the content to be projected. If no is determined in step 301, proceed to step 303: Encode and wirelessly project the content to be projected based on the original bitrate.

[0145] Specifically, the execution subject of this screen projection method can be the screen projection source device. Step 301 can be the same as step 107 above, step 303 can be the same as step 108 above, and step 302 is similar to step 105 above. The difference is that step 303 does not restrict the method of determining the target bitrate. The target bitrate determination method in the above embodiment can be applied, or other possible target bitrate determination methods can be applied, as long as the target bitrate is lower than the original bitrate.

[0146] The screen mirroring method in this embodiment predicts the probability of a user switching to the foreground application in the near future based on the user's historical operation behavior. If the probability of switching to the foreground application is high, bitrate adjustment is less likely, because a decrease in bitrate in the switched foreground application would significantly impact the user's subjective experience and potentially degrade the user experience. Conversely, if the probability of switching is low, bitrate adjustment is more likely, as a decrease in bitrate has a smaller perceptual impact on the user in a stable scenario and is less likely to negatively affect the user experience. Dynamic bitrate adjustment based on the switching frequency of foreground applications can improve user experience while reducing power consumption. Furthermore, the screen mirroring method in this embodiment does not require waiting for the temperature to rise to a threshold before adjusting the bitrate; instead, it dynamically adjusts the bitrate based on the switching frequency of foreground applications. This mitigates the temperature rise of the source device caused by screen mirroring, extends the temperature rise time, and reduces the probability of device malfunctions due to overheating.

[0147] In one possible implementation, step 301, determining whether to perform bitrate adjustment based on the frequency with which the current foreground application switches from the background to the foreground within a preset time period, includes: determining whether to perform bitrate adjustment based on the frequency with which the current foreground application switches from the background to the foreground within the preset time period and a second application type weight. The second application type weight is the application type weight corresponding to the target application that is switched to the foreground within the preset time period, and the target application is an application other than the current foreground application. For example, if the target application that the user has recently switched to is a video application, its corresponding application type weight is relatively large, so bitrate adjustment is more likely; if the target application that the user has recently switched to is, for example, an instant messaging application, its corresponding application type weight is relatively small, so bitrate adjustment is more likely not to be performed. Combining the switching frequency of the foreground application and the application type of the target application to determine whether to perform bitrate adjustment can further improve the user experience.

[0148] In one possible implementation, the application type weight corresponding to a video or game application is greater than the application type weight corresponding to other applications.

[0149] In one possible implementation, determining whether to perform bitrate adjustment based on the frequency of the current foreground application switching from the background to the foreground within a preset time period and the weight of the second application type includes: determining a bitrate adjustment index based on the frequency of the current foreground application switching from the background to the foreground within a preset time period and the weight of the second application type; if the necessity of bitrate adjustment represented by the bitrate adjustment index meets the adjustment conditions, then it is determined to perform bitrate adjustment; if the necessity of bitrate adjustment represented by the bitrate adjustment index does not meet the adjustment conditions, then it is determined not to perform bitrate adjustment. The necessity of bitrate adjustment represented by the bitrate adjustment index is positively correlated with the weight of the second application type, and the necessity of bitrate adjustment represented by the bitrate adjustment index is negatively correlated with the frequency. In other words, the higher the frequency of switching from the background to the foreground within a preset time period, the worse the stability of the scene to be cast, and therefore the lower the necessity for bitrate adjustment. Conversely, the lower the switching frequency, the higher the stability of the scene to be cast, and therefore the higher the necessity for bitrate adjustment. The higher the application type weight of the target application, the greater the probability that the scene to be cast has a high bitrate, and therefore the higher the necessity for bitrate adjustment. Conversely, the lower the application type weight of the target application, the lower the probability that the scene to be cast has a high bitrate, and therefore the lower the necessity for bitrate adjustment.

[0150] In one possible implementation, step 303, encoding the content to be projected and wirelessly projecting it based on the target bitrate, includes: determining the target resolution of the content to be projected based on the target bitrate, wherein the target resolution of the content to be projected is lower than the original resolution of the content to be projected; and encoding and wirelessly projecting the content to be projected based on the target resolution. This approach can minimize the negative impact on user experience while reducing the bitrate. The specific process and principle are the same as in the above embodiments and will not be repeated here.

[0151] This application also provides an electronic device, including a processor and a memory. The memory stores at least one instruction, which, when loaded and executed by the processor, causes the electronic device to perform the method of any of the above embodiments. The specific process and principle of this method are the same as those in the above embodiments, and will not be repeated here. Specifically, this electronic device can... Figure 2 The electronic device shown.

[0152] The electronic devices involved in this application may be any product such as smart TVs, mobile phones, tablets, personal computers (PCs), personal digital assistants (PDAs), smartwatches, wearable electronic devices, augmented reality (AR) devices, virtual reality (VR) devices, in-vehicle devices, drone devices, smart cars, smart speakers, robots, smart glasses, etc.

[0153] This application also provides a computer-readable storage medium, including a program or instructions, wherein the methods in any of the above embodiments are executed when the program or instructions are run on a computer.

[0154] This application also provides a computer program product containing executable instructions that, when executed on a computer, cause the computer to perform the methods described in any of the above embodiments.

[0155] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive).

[0156] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0157] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A screen projection method, characterized in that, include: Obtain the first application type weight, which is the application type weight corresponding to the current foreground application; Whether to perform bitrate adjustment is determined based on bitrate adjustment parameters, including the weight of the first application type. If yes, the content to be cast is encoded and wirelessly cast based on the target bitrate, which is lower than the original bitrate of the content to be cast. If no, the content to be cast is encoded and wirelessly cast based on the original bitrate. The step of determining whether to perform bitrate adjustment based on bitrate adjustment parameters includes: If the weight of the first application type is greater than the preset weight, then it is determined to perform bitrate adjustment; if the weight of the first application type is not greater than the preset weight, then it is determined not to perform bitrate adjustment. Alternatively, a bitrate adjustment index can be determined based on the bitrate adjustment parameters. If the necessity of bitrate adjustment represented by the bitrate adjustment index meets the adjustment conditions, then it is determined to perform bitrate adjustment. If the necessity of bitrate adjustment represented by the bitrate adjustment index does not meet the adjustment conditions, then it is determined not to perform bitrate adjustment. The necessity of bitrate adjustment represented by the bitrate adjustment index is positively correlated with the weight of the first application type. The application type weight is positively correlated with the power consumption benefit of reducing bitrate when the application is cast as a foreground application.

2. The method according to claim 1, characterized in that, The application type weight for video or game applications is greater than the application type weight for other applications.

3. The method according to claim 1, characterized in that, The bitrate adjustment parameters also include the frequency at which the current foreground application switches from the background to the foreground within a preset time period; The necessity of rate adjustment, as represented by the rate adjustment index, is negatively correlated with the frequency.

4. The method according to claim 3, characterized in that, The bitrate adjustment parameter also includes a second application type weight, which is the application type weight corresponding to the target application that is switched to the foreground application during the preset time period. The target application is an application other than the current foreground application.

5. The method according to any one of claims 1 to 4, characterized in that, The process of encoding and wirelessly projecting the content to be projected based on the target bitrate includes: The target resolution of the content to be projected is determined based on the target bitrate, and the target resolution of the content to be projected is lower than the original resolution of the content to be projected. The content to be projected is encoded and wirelessly projected based on the target resolution.

6. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store at least one instruction that, when loaded and executed by the processor, causes the electronic device to perform the method as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, Includes a program or instructions that, when run on a computer, execute the method as described in any one of claims 1 to 5.

8. A computer program product, characterized in that, The computer program product includes executable instructions that, when executed on a computer, cause the computer to perform the method described in any one of claims 1 to 5.

Images