Screen projection method and device, and electronic device

Abstract

This application discloses a screen projection method, apparatus, and electronic device, belonging to the field of screen projection technology. The screen projection method is executed by a first device and includes: acquiring a first image and content parameters of the first image; encoding the first image according to an encoding strategy corresponding to the content parameters to obtain encoded image data; and sending a data frame to a second device, wherein the data frame includes at least the encoded image data.

Description

Technical Field

[0001] This application belongs to the field of screen projection technology, specifically relating to a screen projection method and device, and an electronic device. Background Technology

[0002] With the development of communication technology, screen sharing between devices is becoming increasingly widespread. Examples include screen sharing for meeting interfaces, video conferencing, and game sharing.

[0003] However, existing screen mirroring methods typically employ fixed procedures, leading to suboptimal resource allocation. This can result in insufficient bandwidth, disrupted data transmission, and screen stuttering under the same network bandwidth, often causing mirroring failures and preventing the platform from achieving its optimal rendering performance. Furthermore, due to significant differences in hardware performance across different platforms, such as CPU (Central Processing Unit) and GPU (Graphics Processing Unit) performance, screen specifications (resolution, aspect ratio, color gamut), graphics APIs (Application Programming Interfaces), and system permissions, a single screen mirroring solution cannot leverage the optimal performance of each platform.

[0004] In other words, in related technologies, there is a lack of dynamic adaptive capability for screen projection between devices when facing diverse hardware platforms, operating systems and complex application scenarios. Summary of the Invention

[0005] The purpose of this application is to provide a screen projection method, device, and electronic device that can improve the dynamic adaptive capability of screen projection between devices.

[0006] In a first aspect, embodiments of this application provide a screen projection method executed by a first device. The screen projection method includes: acquiring a first image and content parameters of the first image; encoding the first image according to an encoding strategy corresponding to the content parameters to obtain encoded image data; and sending a data frame to a second device, wherein the data frame includes at least the encoded image data.

[0007] Secondly, this application provides another screen projection method, executed by a second device. The screen projection method includes: receiving a data frame sent by a first device, the data frame including at least encoded image data of a first image; obtaining the encoded image data and content parameters of the first image; and decoding and rendering the encoded image data using a decoding and rendering strategy corresponding to the content parameters to obtain a second image.

[0008] Thirdly, embodiments of this application provide a screen projection device applied to a first device. The screen projection device includes: a first processing unit, configured to acquire a first image and content parameters of the first image; the first processing unit is further configured to encode the first image according to an encoding strategy corresponding to the content parameters to obtain encoded image data; and a first communication unit, configured to send a data frame to a second device, wherein the data frame includes at least the encoded image data.

[0009] Fourthly, this application provides another screen projection device applied to a second device. The screen projection device includes: a second communication unit for receiving data frames sent by a first device, wherein the data frames include at least encoded image data of a first image; a second processing unit for acquiring the encoded image data and content parameters of the first image; and the second processing unit is further configured to decode and render the encoded image data using a decoding and rendering strategy corresponding to the content parameters to obtain a second image.

[0010] Fifthly, embodiments of this application provide an electronic device, which includes a processor and a memory. The memory stores programs or instructions that can run on the processor. When the program or instructions are executed by the processor, they implement the steps of the screen projection method as described in the first aspect, and / or, when the program or instructions are executed by the processor, they implement the steps of the screen projection method as described in the second aspect.

[0011] Sixthly, embodiments of this application provide a readable storage medium storing a program or instructions, which, when executed by a processor, implement the steps of the screen projection method as described in the first aspect, and / or, when executed by a processor, implement the steps of the screen projection method as described in the second aspect.

[0012] In a seventh aspect, embodiments of this application provide a chip, which includes a processor and a communication interface, the communication interface and the processor being coupled, the processor being used to run programs or instructions to implement the steps of the screen projection method as described in the first aspect, and / or the processor being used to run programs or instructions to implement the steps of the screen projection method as described in the second aspect.

[0013] Eighthly, embodiments of this application provide a computer program product, which is stored in a storage medium and executed by at least one processor to implement the steps of the screen projection method as described in the first aspect, and / or, the computer program product is executed by at least one processor to implement the steps of the screen projection method as described in the second aspect.

[0014] In the screen projection method provided in this application embodiment, for the first device, a first image and its content parameters are obtained; the first image is encoded according to the encoding strategy corresponding to the content parameters to obtain encoded image data; a data frame is sent to the second device, the data frame including at least the encoded image data. Through the above screen projection method, during the projection process, at the projection end, based on the content parameters of the first image to be projected, a corresponding encoding strategy is selected to encode the first image. This achieves adaptive selection of the encoding strategy based on the content parameters, optimizes resource allocation, and improves the dynamic adaptive capability of screen projection between devices. Attached Figure Description

[0015] Figure 1 One of the flowcharts of the screen projection method provided in the embodiments of this application;

[0016] Figure 2 The second schematic flowchart of the screen projection method provided in the embodiments of this application;

[0017] Figure 3 The third flowchart illustrating the screen projection method provided in this application embodiment;

[0018] Figure 4 A schematic diagram illustrating the screen projection method provided in this application embodiment;

[0019] Figure 5 The fourth flowchart illustrating the screen projection method provided in this application embodiment;

[0020] Figure 6 Fifth flowchart illustrating the screen projection method provided in this application embodiment;

[0021] Figure 7 A flowchart illustrating the screen projection method provided in this application embodiment is shown in Figure 6.

[0022] Figure 8 The seventh flowchart illustrating the screen projection method provided in this application embodiment;

[0023] Figure 9 This is one of the structural block diagrams of the screen projection device provided in the embodiments of this application;

[0024] Figure 10 This is a second structural block diagram of the screen projection device provided in the embodiments of this application;

[0025] Figure 11 A structural block diagram of the electronic device provided in the embodiments of this application;

[0026] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0028] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0029] The screen projection method, apparatus, and electronic device provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0030] like Figure 1 As shown, this application embodiment provides a screen mirroring method, executed by a first device, which may specifically include the following steps S102 to S106:

[0031] S102: Obtain the first image and its content parameters.

[0032] The screen projection method proposed in this application is executed by a first device, which may be an electronic device such as a smartphone, tablet computer, or smartwatch, and is not specifically limited here.

[0033] The first image is the screen image of the first device.

[0034] Optionally, the content parameter is used to indicate the image content features of the first image.

[0035] Optionally, the content parameter may specifically be the image content type of the first image. The image content type is used to indicate the frame rate of the first image, that is, to indicate how fast the content of the screen image of the first device is updated.

[0036] Optionally, the content parameter can also be specifically the image content blocks of the first image. The image content blocks are used to indicate the degree of user attention to different areas in the first image, such as user-focused areas and non-focused areas.

[0037] In practical applications, the above-mentioned parameters can be determined by the first device based on the image parameters of the first image, or they can be sent by the second device to the first device; no specific restrictions are imposed here.

[0038] The second device can be any electronic device such as a tablet computer, a laptop computer, or a smart TV, without any specific restrictions.

[0039] S104: According to the encoding strategy corresponding to the content parameters, the first image is encoded to obtain encoded image data.

[0040] Specifically, in the screen projection method provided in this application embodiment, during the projection process, for the first device at the projection end, a first image is obtained by capturing a screen image, and the content parameters of the first image are acquired. Based on this, the first device then encodes the first image according to the encoding strategy corresponding to the content parameters to obtain encoded image data. In this way, a projection process of "content recognition → classification encoding" is realized at the projection end. Compared with the traditional fixed projection process, this optimizes the allocation of encoding resources and reduces system load and transmission latency while ensuring projection effectiveness.

[0041] S106: Send a data frame to the second device.

[0042] The data frame includes at least coded image data.

[0043] Specifically, in the screen projection method provided in this application embodiment, after the first device encodes the first image to obtain encoded image data, the first device sends a data frame containing at least the encoded image data to the second device, so that the second device can obtain the second image by decoding and rendering the encoded image data.

[0044] In practical applications, the data frame can also include content parameters of the first image, enabling the second device to decode and render the encoded image data using the decoding and rendering strategy corresponding to the content parameters, thereby obtaining the second image. This achieves a screen projection process of "content recognition → classification encoding → platform-adaptive rendering." Compared to traditional fixed screen projection processes, this optimizes resource allocation, reduces system load and transmission latency, and adapts to differences between various hardware platforms while ensuring projection quality, thus balancing high performance, low latency, high quality, and universality of screen projection between devices.

[0045] The screen projection method provided in this application, for a first device, involves acquiring a first image and its content parameters; encoding the first image according to an encoding strategy corresponding to the content parameters to obtain encoded image data; and sending a data frame to a second device, the data frame including at least the encoded image data. Through this screen projection method, during the projection process, the projection end selects a corresponding encoding strategy to encode the first image based on its content parameters. This achieves adaptive selection of the encoding strategy based on the content parameters, optimizes the allocation of encoding resources, and enhances the dynamic adaptive capability of screen projection between devices.

[0046] In this embodiment of the application, the content parameter includes image content type, which is used to indicate the frame rate range of the first image. Based on this, the above-mentioned S104 may specifically include the following S104a and S104b:

[0047] S104a: When the frame rate of the first image is less than or equal to the preset frame rate, the entire region of the first image in the first frame is encoded. Starting from the first image in the second frame, only the regions in the first image that differ from the previous frame are encoded to obtain the encoded image data corresponding to the first image in each frame.

[0048] Specifically, in the screen projection method provided in this application embodiment, when the frame rate of the first image is less than or equal to a preset frame rate, for the first frame of the first image, the first device encodes the entire region of the first image to obtain the encoded image data corresponding to the first frame of the first image. Starting from the second frame of the first image, the first device continuously compares the two frames of the first image, detects and extracts the parts of the image content that have changed between the current first image and the previous frame of the first image, obtains the difference regions in the current first image and the previous frame of the first image, and only encodes the extracted difference regions to obtain the encoded image data corresponding to each frame of the first image. For areas where the image content has not changed, such as background and text, the data of the previous frame of the first image is directly reused.

[0049] The first device determines the image content type of the first image by detecting whether the frame rate of the screen image is greater than a preset frame rate when capturing the screen image. Specifically, if the frame rate of the screen image is detected to be greater than the preset frame rate, the first image is defined as a dynamic image; otherwise, the first image is defined as a static image. After determining the image content type of the first image, the first device writes a type flag into the header of each data frame for transmission to the second device.

[0050] The specific value of the preset frame rate can be set by those skilled in the art according to the actual situation. For example, the preset frame rate is 24fps, and no specific limit is made here.

[0051] S104b: When the frame rate of the first image is greater than the preset frame rate, the entire region of the first image in each frame is encoded to obtain the encoded image data corresponding to the first image in each frame.

[0052] Specifically, in the screen projection method provided in this application embodiment, when the frame rate of the first image is greater than the preset frame rate, the screen changes rapidly and the difference between the previous and subsequent frames is large. Therefore, the first device encodes the entire area of ​​the first image in each frame to obtain the encoded image data corresponding to the first image in each frame.

[0053] In practical applications, the first device can use video coding standards such as H.264 / H.265 to perform high compression on dynamic images.

[0054] Based on this, the first device merges the encoded image data of each frame of the first image and the corresponding image content type into a data frame, and sends the data frame to the second device, so that the second device can adaptively select a decoding and rendering strategy based on the image content type to decode and render the encoded image data.

[0055] That is, in the screen projection method provided in this application embodiment, the first device dynamically adjusts the encoding strategy according to the image content type, and the second device dynamically adjusts the rendering strategy according to the image content type. Specifically, for static images, the strategy of "detecting differences + incremental update" is selected to save bandwidth first, and for dynamic images, the strategy of "full frame encoding + hardware decoding and hardware display" is selected to ensure real-time performance first, thereby achieving efficient and low-latency screen projection effect.

[0056] For example, such as Figure 2 As shown, for the first device, the screen projection method provided in this application embodiment may specifically include the following S108 to S126:

[0057] S108: Determine if the frame rate is greater than 24fps. If yes, proceed to S110; otherwise, proceed to S114.

[0058] S110: Define the first image as a dynamic image.

[0059] S112: Encode the entire region of the first image.

[0060] S114: Define the first image as a static image.

[0061] S116: Compare the differences between the previous and next frames.

[0062] S118: Extract the difference region.

[0063] S120: Encoding of the difference region.

[0064] S122: High compression using the H.264 / H.265 video coding standard.

[0065] S124: Write the image content type to the header of each data frame.

[0066] S126: Send the data frame to the second device.

[0067] The embodiments provided in this application include image content type as a content parameter, which indicates the frame rate range of the first image. When the frame rate of the first image is less than or equal to a preset frame rate, the entire region of the first image in the first frame is encoded. Starting from the second image, only the regions in the first image that differ from the previous frame are encoded, resulting in encoded image data corresponding to each first image frame. When the frame rate of the first image is greater than the preset frame rate, the entire region of each first image frame is encoded, resulting in encoded image data corresponding to each first image frame. This achieves content-based classification encoding. Different compression qualities are selected for first images with different content types, allowing for selective encoding and transmission. This reduces the amount of data that needs to be transmitted, thereby saving available transmission bandwidth, reducing rendering pressure, and decreasing the frequency of adaptive rendering. This facilitates the use of efficient local rendering APIs to complete rendering, ensuring smooth screen projection.

[0068] In this embodiment, the content parameters include image content blocks, and the image content blocks include user-focused and non-focused areas; based on this, the above-mentioned S104 may specifically include the following S104c and S104d:

[0069] S104c: Divide the user-focused and non-focused regions in the first image.

[0070] Specifically, in the screen projection method provided in this application embodiment, the second device can analyze the projection interface to determine the user-focused and non-focused areas of the projection interface, that is, determine the image content blocks of the first image, and feed back the determined image content blocks to the first device. The first device receives the image content blocks sent by the second device, and divides the user-focused and non-focused areas in the first image according to the indication of the image content blocks.

[0071] S104d: Encode the user-interested region in each frame of the first image according to the first compression ratio and the first frame rate, and encode the non-interested region in each frame of the first image according to the second compression ratio and the second frame rate to obtain the encoded image data corresponding to each frame of the first image.

[0072] The first compression ratio is lower than the second compression ratio. The smaller the compression ratio, the lighter the compression and the better the image quality; the larger the compression ratio, the more severe the compression and the smaller the data volume.

[0073] Optionally, the first frame rate is greater than the second frame rate.

[0074] Specifically, in the screen projection method provided in this application embodiment, after dividing the user-focused area and non-focused area in the first image, for the user-focused area, the first device encodes the user-focused area in each frame of the first image at a lower first compression ratio, and transmits the encoded image data corresponding to the user-focused area in each frame of the first image at a higher first frame rate, so as to ensure the clarity and real-time performance of the screen projection of the user-focused area, while taking into account the low latency and high quality effect of screen projection between devices. For the non-focused area, the first device encodes the non-focused area in each frame of the first image at a higher second compression ratio, and transmits the encoded image data corresponding to the non-focused area in each frame of the first image at a lower second frame rate, so as to save more than 50% of bandwidth without affecting the basic screen projection effect, thereby achieving fast screen projection.

[0075] In practical applications, the first device can also transmit the encoded image data corresponding to the user's area of ​​interest in advance or separately, without specific restrictions.

[0076] The embodiments provided in this application include image content segmentation, where each segment comprises user-focused and non-focused regions. The first image is divided into user-focused and non-focused regions. The user-focused regions in each frame of the first image are encoded according to a first compression ratio and a first frame rate, and the non-focused regions in each frame of the first image are encoded according to a second compression ratio and a second frame rate, resulting in encoded image data corresponding to each frame of the first image. The first compression ratio is less than the second compression ratio, and the first frame rate is greater than the second frame rate. This achieves spatially differentiated encoding based on image content segmentation, optimizes resource allocation, and saves bandwidth while maintaining low latency and high-quality projection between devices, enabling fast projection.

[0077] like Figure 3 As shown, this application embodiment provides another screen projection method, executed by a second device. The screen projection method may specifically include the following steps S202 to S206:

[0078] S202: Receive the data frame sent by the first device.

[0079] The screen projection method proposed in this application is executed by a second device, which may be an electronic device such as a tablet computer, a laptop computer, or a smart TV, and is not specifically limited here.

[0080] The first device can be any electronic device such as a smartphone, tablet, or smartwatch, without any specific restrictions.

[0081] Optionally, the data frame includes at least the encoded image data of the first image, and may also include content parameters of the first image.

[0082] S204: Obtain the encoded image data and the content parameters of the first image.

[0083] The encoded image data is obtained by the first device encoding the first image according to the encoding strategy corresponding to the content parameters.

[0084] Optionally, the content parameters of the first image can be obtained by parsing the data frame, or determined by the second device by analyzing the projection interface, and pre-stored in the second device.

[0085] Specifically, in the screen projection method provided in this application embodiment, during the screen projection process, for the second device at the screen projection end, the second device can receive data frames sent by the first device, and obtain the encoded image data of the first image by parsing the data frames, and obtain the content parameters of the first image by parsing the data frames, or directly retrieve the pre-stored content parameters.

[0086] S206: Decode and render the encoded image data using the decoding and rendering strategy corresponding to the content parameters to obtain the second image.

[0087] The second image is the image displayed on the projection interface, and the content of the second image is the same as that of the first image.

[0088] Specifically, in the screen projection method provided in this application embodiment, after obtaining the encoded image data of the first image and the corresponding content parameters, the second device adaptively selects the corresponding decoding and rendering strategy based on the content parameters to decode and render the encoded image data to obtain the second image.

[0089] The screen projection method provided in this application, for a second device, receives a data frame sent by a first device, the data frame including at least encoded image data of a first image; obtains the encoded image data and content parameters of the first image; and decodes and renders the encoded image data using a decoding and rendering strategy corresponding to the content parameters to obtain a second image. Through this screen projection method, during the projection process, on the receiving end, a corresponding decoding and rendering strategy is selected based on the content parameters of the first image to be projected for decoding and rendering operations. This achieves adaptive selection of decoding and rendering strategies based on content parameters, optimizes rendering resource allocation, improves the dynamic adaptive capability of screen projection between devices, and can adapt to the differences between different hardware platforms, thus balancing high performance, low latency, high quality, and universality of screen projection between devices.

[0090] In this embodiment of the application, the content parameter includes the image content type. Based on this, the above-mentioned S206 may specifically include the following S206a to S206c:

[0091] S206a: Based on the image content type and the operating system type of the second device, determine candidate strategies for decoding and rendering encoded image data.

[0092] The data frame includes image content types.

[0093] Optionally, the operating system type is used to indicate the hardware platform type of the second device, such as Windows platform, macOS platform, etc.

[0094] Specifically, in the screen projection method provided in this application embodiment, the second device parses the data frame to obtain the encoded image data and the corresponding image content type. Then, it uses the image content type as a "decision switch" for subsequent decoding and rendering processing schemes to activate the optimal decoding and rendering pipeline, balancing the image quality, latency, and performance of the projection. Specifically, the second device determines candidate strategies for decoding and rendering the encoded image data based on the image content type and its own operating system type. In this way, by making a first-level decision based on the image content type when selecting a decoding and rendering strategy, resources can be pre-configured to prepare the decoder and rendering backend for the upcoming data stream, while preventing the use of high-overhead solutions for processing dynamic images to handle simple static image data streams, thus reducing performance waste.

[0095] For dynamic images, decoding and rendering operations aim for low latency, while for static images, decoding and rendering operations aim for high efficiency.

[0096] Specifically, for rendering static images, the real-time requirements are relatively low, but clarity and tear-free rendering are required. The priority of choosing a rendering scheme can be as follows:

[0097] Direct3D (a 3D graphics API): The preferred solution for Windows platforms, it can efficiently copy and render 2D bitmaps using modern GPUs, with extremely high efficiency; Metal (a graphics API): The preferred solution for macOS platforms, it can efficiently copy and render 2D bitmaps using modern GPUs, with extremely high efficiency; OpenGL (a cross-platform open-source graphics API): As a cross-platform alternative to ensure compatibility; Pure software rendering: As a last resort, it is rendered by the CPU when the GPU is unavailable.

[0098] For dynamic images, the requirements for real-time performance and efficiency in decoding and rendering are high. The priority of rendering scheme selection can be as follows:

[0099] Windows platform: Uses Direct3D, which can directly interface with the hardware decoder. The decoded video data can be sent to the GPU for rendering and display with "zero copying", resulting in the shortest strategy and lowest latency. macOS or iOS platform: Uses Metal, which enables efficient transfer from hardware decoding to Metal textures. Cross-platform or legacy system: Uses OpenGL, a universal backup solution that supports uploading and rendering video frames via textures. Pure software rendering: Only used as a last resort when no GPU acceleration can be used, it draws pixels through the CPU.

[0100] S206b: Determine the decoding rendering strategy from the list of available strategies supported by the second device, based on the candidate strategies.

[0101] The available strategy list refers to the list of rendering schemes that the second device can support.

[0102] Optionally, the list of available strategies includes the optimal strategy that supports hardware decoding and hardware rendering, a compatible strategy for general GPU rendering, and a safety net strategy without GPU acceleration, so as to enable the selection of strategies that reduce performance.

[0103] S206c: Decode and render the encoded image data according to the decoding and rendering strategy to obtain the second image.

[0104] Specifically, in the screen projection method provided in this application embodiment, after determining candidate strategies for decoding and rendering encoded image data, the second device determines a decoding and rendering strategy from its own list of supported available strategies based on the candidate strategies, and decodes and renders the encoded image data according to the selected decoding and rendering strategy to obtain a second image. In this way, when selecting a decoding and rendering strategy, a second-level decision is made by combining the second device's list of available strategies, selecting the optimal solution from the actually available technologies. It does not need to consider the second device's graphics card type or whether the graphics card supports hardware decoding and rendering; it can automatically select the correct and available strategy for screen projection, demonstrating strong adaptability.

[0105] That is, in the screen mirroring method provided in the embodiments of this application, such as Figure 4 As shown, when selecting a decoding and rendering strategy, a first-level decision is made based on the image content type corresponding to the encoded image data transmitted from the first device to the second device. Then, a second-level decision is made by combining the list of available strategies related to the graphics card configuration. Thus, based on the dynamic dual-decision logic of content-driven and resource-adaptive, the rendering scheme is dynamically determined to achieve highly compatible and highly smooth screen projection.

[0106] The embodiments provided in this application include content parameters such as image content type. Based on the image content type and the operating system type of the second device, candidate strategies for decoding and rendering the encoded image data are determined. Based on the candidate strategies, a decoding and rendering strategy is determined from the list of available strategies supported by the second device. The encoded image data is then decoded and rendered according to the decoding and rendering strategy to obtain a second image. This achieves a dynamic dual-decision logic of content-driven and resource-adaptive approaches. Facing diverse graphics cards and driver configurations, it can adaptively select decoding and rendering strategies, thereby solving the performance problems caused by general-purpose cross-platform rendering technologies. It has strong adaptability and can leverage the unique rendering advantages of different platforms.

[0107] In this embodiment of the application, prior to S206b, the screen projection method further includes the following S208 and S210:

[0108] S208: Start the detection service program, detect the feasibility of the hardware decoder and different rendering application interfaces, and output the detection results.

[0109] Specifically, in the screen projection method provided in this application embodiment, after the second device is started, it can perform a power-on self-test. Before screen projection, a background service program, namely a detection service program, is started to execute hardware pre-detection logic to detect whether the second device supports hardware decoding and hardware rendering.

[0110] Specifically, before screen mirroring, the second device starts a detection service program to automatically detect the hardware decoding and hardware rendering support of the graphics card driver. Specifically, it detects the feasibility of hardware decoders (such as H.264 / H.265 decoders) and different rendering application interfaces (such as Direct3D, Metal, and OpenGL), and outputs the detection results.

[0111] The detection results can be presented in the form of a detection report. That is, the second device generates a system capability "white paper" through preprocessing and self-discovery, which informs the second device of its support for different hardware decoders and rendering application interfaces.

[0112] S210: Determine the list of available strategies based on the detection results.

[0113] Specifically, in the screen casting method provided in this application embodiment, after starting the detection service program and outputting the detection results, the second device determines the list of available strategies based on the detection results, and stores the list of available strategies as a configuration file in the local memory of the second device, so that the second device can automatically select the available decoding and rendering strategies for screen casting based on the list of available strategies.

[0114] For example, such as Figure 5As shown, the second device's process for detecting the operability of the hardware decoder and different rendering application programming interfaces may specifically include the following steps S224 to S246:

[0115] S224: Start hardware pre-detection.

[0116] S226: Determine if a hardware decoding error occurs. If yes, proceed to S230; otherwise, proceed to S228.

[0117] S228: Hardware decoding.

[0118] S230: CPU software decoding.

[0119] S232: Determine if a hardware rendering error occurs. If yes, proceed to S246; otherwise, proceed to S234.

[0120] S234: Determine if Windows platform is used. If yes, proceed to S236; otherwise, proceed to S238.

[0121] S236: Direct3D rendering.

[0122] S238: Metal rendering.

[0123] S240: Determine if rendering is successful. If yes, end the process; otherwise, execute S242.

[0124] S242: OpenGL rendering.

[0125] S244: Determine if rendering is successful. If yes, end the process; otherwise, execute S246.

[0126] S246: Pure software rendering.

[0127] In practical applications, the second device can also read the graphics card information when screen mirroring is started, and determine whether the second device supports hardware decoding and hardware rendering based on the graphics card information, thereby generating a list of available strategies.

[0128] For example, such as Figure 6 As shown, the process by which the second device determines the operability of the hardware decoder and different rendering application interfaces may specifically include the following steps S248 to S268:

[0129] S248: Read graphics card information.

[0130] S250: Determine if hardware decoding is supported. If yes, proceed to S254; otherwise, proceed to S252.

[0131] S252: CPU software decoding.

[0132] S254: Determine if hardware rendering is supported. If yes, proceed to S258; otherwise, proceed to S256.

[0133] S256: Pure software rendering.

[0134] S258: Determine if Windows platform is used. If yes, proceed to S260; otherwise, proceed to S262.

[0135] S260: Direct3D rendering.

[0136] S262: Metal rendering.

[0137] S264: Determine if a hardware rendering error occurs. If yes, proceed to S266; otherwise, terminate the process.

[0138] S266: OpenGL rendering.

[0139] S268: Determine if rendering is successful. If yes, end the process; otherwise, execute S256.

[0140] Based on this, for example, such as Figure 7 As shown, for the second device, the screen projection method provided in this application embodiment may specifically include the following S302 to S328:

[0141] S302: Second device started.

[0142] S304: Detects the decoding and rendering capabilities of the graphics card driver.

[0143] S306: Output the detection results.

[0144] S308: Establish a list of available strategies based on the detection results.

[0145] S310: Standby, waiting for screen mirroring signal.

[0146] S312: Receives screen projection signals and parses image content types.

[0147] S314: Intelligent selection based on image content type and a list of available strategies.

[0148] S316: For moving images, select the strategy with the lowest latency from the list of available strategies.

[0149] S318: When hardware decoding is supported, it can be rendered through Direct3D or Metal hardware decoding and hardware display channels.

[0150] S320: When hardware decoding is not supported, rendering is performed via OpenGL or software decoding.

[0151] S322: For static images, select the most efficient strategy from the list of available strategies.

[0152] S324: When GPU support is available, it decodes and renders through Direct3D, Metal, or OpenGL channels.

[0153] S326: Pure software rendering when there is no GPU support.

[0154] S328: Complete image rendering.

[0155] Therefore, the screen projection method provided in this application embodiment offers a performance degradation strategy selection scheme from Direct3D / Metal hardware decoding and hardware display to OpenGL, and then to pure software rendering. This ensures that a usable decoding and rendering strategy can be found in any hardware environment, thus ensuring the robustness and practicality of screen projection between devices.

[0156] The embodiments provided in this application initiate a detection service program to detect the operability of the hardware decoder and different rendering application programming interfaces, and output the detection results; based on the detection results, a list of available strategies is determined. This facilitates broad adaptability of screen projection across platforms and hardware. Facing diverse graphics cards and driver configurations, it can adaptively select decoding and rendering strategies, thereby solving the performance problems caused by general-purpose cross-platform rendering technologies. It has strong adaptability and can leverage the unique rendering advantages of different platforms.

[0157] In this embodiment of the application, the content parameters include image content blocks, and the image content blocks include user-focused and non-focused regions. Based on this, the above-mentioned S206 may specifically include the following S206d:

[0158] S206d: Decode and render the encoded image data corresponding to the user's area of ​​interest according to the first rendering strategy, and decode and render the encoded image data corresponding to the non-area of ​​interest according to the second rendering strategy to obtain the second image.

[0159] The latency of the first rendering strategy is less than that of the second rendering strategy.

[0160] Specifically, in the screen projection method provided in this application embodiment, the second device can analyze the projection interface to determine the user-focused and non-focused areas of the projection interface, that is, to determine the image content blocks of the first image, and feed back the determined image content blocks to the first device. Based on this, after receiving and parsing the data frames sent by the first device, for the encoded image data corresponding to the user-focused area, the second device selects the first rendering strategy with the lowest latency and highest performance from the available strategy list to decode and render the encoded image data. For example, for the Windows platform, the hardware decoding and direct rendering pipeline of Direct3D is selected to ensure that the interaction latency of the user-focused area is minimized.

[0161] For encoded image data corresponding to non-interested areas, the second device can flexibly choose a more compatible second rendering strategy from the available strategy list, such as OpenGL pipeline or software rendering, to decode and render the encoded image data, so as to give way to GPU core resources for processing the areas of interest to the user.

[0162] The embodiments provided in this application include image content blocks as content parameters. Each image content block includes user-focused and non-focused regions. The encoded image data corresponding to the user-focused regions is decoded and rendered according to a first rendering strategy, and the encoded image data corresponding to the non-focused regions is decoded and rendered according to a second rendering strategy to obtain a second image. The latency of the first rendering strategy is less than the latency of the second rendering strategy. This achieves spatially differentiated rendering based on image content blocks, optimizes resource allocation, prioritizes system resources for processing user-focused regions, and improves resource utilization efficiency. It can save bandwidth and improve the real-time performance of inter-device projection while maintaining low latency and high-quality effects. Furthermore, it represents a revolution in inter-device projection from content attributes to visual attention, demonstrating significant practical value in fields sensitive to real-time performance and visual quality, such as remote collaboration, cloud gaming, and in-vehicle projection.

[0163] In this embodiment of the application, before S206, the screen projection method further includes the following S212 to S222:

[0164] S212: Obtain the user's eye tracking information, interaction information, and screen content information on the projection interface.

[0165] Among them, eye-tracking information is used to reflect the user's spontaneous visual attention. Specifically, it can be the signal collected by the front-facing camera on the second device, which can be used for eye tracking. Eye-tracking information is recorded as physiological signals.

[0166] Optionally, the interaction information is used to reflect the user's operation intention. The interaction information includes, but is not limited to: cursor position, touch point position, keyboard focus position, crosshair coordinates in the game, and editing point position in the document, etc. There are no specific limitations here. The interaction information is referred to as the interaction signal.

[0167] Optionally, the screen content information is used to reflect the inherent salient features of the screen content of the projection interface. The screen content information includes, but is not limited to: the position of UI (User Interface) controls, the position of moving objects, and the position of faces in video conferencing, etc. There are no specific limitations here. The screen content information is referred to as semantic signals.

[0168] S214: Determine the coordinates of the user's gaze focus on the projection interface based on human eye tracking information.

[0169] Specifically, in the screen projection method provided in this application embodiment, after obtaining the user's eye tracking information on the screen projection interface, the second device uses a lightweight neural network model to calculate the intersection coordinates of the user's gaze vector and the screen of the second device in real time based on the eye tracking information, thereby obtaining the coordinates of the user's gaze focus on the screen projection interface, and calculating the confidence level of the gaze focus coordinates.

[0170] S216: Determine the user's focus area on the projection interface based on the interaction information.

[0171] Specifically, in the screen projection method provided in this application embodiment, after obtaining the user's interaction information on the screen projection interface, the second device determines the user's interaction focus area on the screen projection interface, such as the operation window area or the operation control area, based on the interaction information.

[0172] S218: Determine the semantic attention area of ​​the projection interface based on the content information of the screen.

[0173] Specifically, in the screen projection method provided in this application embodiment, after obtaining the screen content information of the projection interface, the second device determines the semantic interest area of ​​the projection interface, such as the face frame and the moving object frame, based on the screen content information.

[0174] S220: Based on the coordinates of the gaze focus, the interaction focus area, the semantic attention area, and time information, the different areas of the projection interface are dynamically integrated and weighted.

[0175] Specifically, in the screen projection method provided in this application embodiment, after determining the user's gaze focus coordinates, interactive focus area, and semantic attention area on the projection interface, the second device introduces a configurable weight calculation and arbitration engine. Based on multi-dimensional allocation rules, and according to the gaze focus coordinates, interactive focus area, semantic attention area, and time information, it dynamically fuses and assigns weights to different areas of the projection interface to determine the weights of different areas in the projection interface. This facilitates intelligent decision-making for image content segmentation and improves the scene adaptability of image content segmentation.

[0176] The aforementioned multidimensional allocation rules include at least one of the following or a combination thereof: signal superposition enhancement rules, signal conflict arbitration rules, and time decay mechanism rules.

[0177] The signal superposition reinforcement rule refers to increasing the weight of an area when multiple signals point to the same area. For example, if the user is looking at the cursor's click location, the weight of the area where the cursor is located is increased; or, if the cursor is hovering over a UI control, the weight of the area where the UI control is located is increased.

[0178] Optionally, the signal conflict arbitration rule refers to arbitrating based on the system's built-in priority rules when two signals conflict, selecting the region that best fits the current context. For example, if a user's gaze is on A but they are interacting with B, in a productivity scenario, the interaction intent takes precedence over visual dwell time, and the region where B is located is selected. In a viewing scenario, however, visual dwell time takes precedence over interaction intent, and the region where A is located is selected.

[0179] Optionally, the time decay mechanism rule means that the weight of the old focus decreases over time to ensure that the user's attention area can be smoothly and naturally shifted with the user's behavior, avoiding residual interference from the old area.

[0180] In practical applications, the decision logic of the weight calculation and arbitration engine, as well as the multi-dimensional allocation rules mentioned above, can be optimized through machine learning or user configuration, and no specific restrictions are imposed here.

[0181] S222: Determine at least one user-focused area based on the weight of different areas in the projection interface, and determine the areas in the projection interface other than the user-focused area as non-focused areas.

[0182] Specifically, in the screen projection method provided in this application embodiment, after dynamically merging and weighting different areas of the screen projection interface, the second device determines at least one user-focused area based on the weights of different areas in the screen projection interface, and determines the areas in the screen projection interface other than the user-focused area as non-focused areas.

[0183] Based on this, after determining the user's area of ​​interest, the coordinates and attributes of the user's area of ​​interest are pushed to the encoder and renderer in real time as the basis for resource allocation.

[0184] In practical applications, the determination of the user's attention region is updated in real time at the millisecond level. Specifically, the user's attention region is updated rapidly every frame or every two frames. Alternatively, when the user's gaze or focus shifts, the user's attention region and non-attention region are immediately redefined, and encoding resources and rendering pipelines are dynamically scheduled. In this way, a high-speed closed loop from perception to execution is achieved, enabling on-demand allocation and real-time scheduling of computing and bandwidth resources. This maximizes the value of attention calculation results, directly translating into quantifiable performance improvements, such as reduced latency and bandwidth savings.

[0185] In summary, such as Figure 8 As shown, the screen projection method provided in this application embodiment may specifically include the following S402 to S428:

[0186] S402: Start user attention area analysis.

[0187] S404: Loop to acquire the signal at the next moment.

[0188] S406: Analyze human eye tracking information.

[0189] S408: Analyze the interactive information.

[0190] S410: Analyze the information in the image content.

[0191] S412: Outputs the coordinates and confidence level of the viewpoint focus.

[0192] S414: Output the interactive focus area.

[0193] S416: Output semantically relevant regions.

[0194] S418: Dynamic fusion and weight allocation based on multi-dimensional allocation rules.

[0195] S420: Output at least one area of ​​user interest.

[0196] S422: Pushes the coordinates and attributes of the area of ​​interest to the encoder and renderer in real time.

[0197] S424: Implement a layered coding and rendering strategy: In areas of user interest, use a high-quality coding + high-performance rendering strategy; in areas of non-interest, use a high-performance coding + standard rendering strategy.

[0198] S426: Complete one frame processing.

[0199] S428: Delay by one or two frames.

[0200] This approach enables non-homogeneous dynamic resource allocation guided by visual saliency. Specifically, the parallel acquisition of three complementary signal types forms a stereo perception system, achieving parallel acquisition and fusion of multimodal signals. This ensures stable and reliable attention input across various application scenarios, demonstrating strong robustness. Furthermore, dynamic fusion and weight allocation based on multidimensional allocation rules enable context- and weight-based dynamic decision-making, enhancing the intelligence of user attention region determination. Additionally, the user attention region is updated in real-time at millisecond levels, achieving a rapid closed loop from perception to execution, and enabling on-demand allocation and real-time scheduling of computing and bandwidth resources.

[0201] The embodiments provided in this application acquire user eye-tracking information, interaction information, and screen content information of the projection interface; determine the user's gaze focus coordinates on the projection interface based on the eye-tracking information; determine the user's interaction focus area on the projection interface based on the interaction information; determine the semantic attention area of ​​the projection interface based on the screen content information; dynamically fuse and weight different areas of the projection interface based on the gaze focus coordinates, interaction focus area, semantic attention area, and time information; determine at least one user attention area based on the weights of different areas in the projection interface, and determine the areas of the projection interface other than the user attention area as non-attention areas. Thus, by using parallel acquisition and fusion of multimodal signals, and dynamic decision-making based on context and weights, the user attention area is determined, improving the accuracy of image content segmentation.

[0202] The screen mirroring method provided in this application can be executed by a screen mirroring device. This application uses a screen mirroring device executing the above-described screen mirroring method as an example to illustrate the screen mirroring device provided in this application.

[0203] like Figure 9 As shown, this application embodiment provides a screen projection device 500, which is applied to a first device. The screen projection device 500 can be the first device or a component in the first device, such as an integrated circuit or a chip.

[0204] Specifically, the projection device 500 may include the first processing unit 502 and the first communication unit 504 described below.

[0205] The first processing unit 502 is used to acquire the first image and the content parameters of the first image;

[0206] The first processing unit 502 is further configured to encode the first image according to the encoding strategy corresponding to the content parameters to obtain encoded image data;

[0207] The first communication unit 504 is used to send data frames to the second device, the data frames including at least encoded image data.

[0208] The screen projection device 500 provided in this application embodiment, for a first device, acquires a first image and its content parameters; encodes the first image according to the encoding strategy corresponding to the content parameters to obtain encoded image data; and sends a data frame to a second device so that the second device decodes and renders the encoded image data according to the decoding and rendering strategy corresponding to the content parameters to obtain a second image. Through the screen projection device 500, during the projection process, at the projection end, based on the content parameters of the first image to be projected, a corresponding encoding strategy is selected to encode the first image. This achieves adaptive selection of the encoding strategy based on the content parameters, optimizes the allocation of encoding resources, and improves the dynamic adaptive capability of screen projection between devices.

[0209] In this embodiment, the content parameter includes image content type, which is used to indicate the frame rate range of the first image. The first processing unit 502 is specifically used to: encode all regions of the first frame of the first image when the frame rate of the first image is less than or equal to a preset frame rate; starting from the second frame of the first image, only the regions in the first image that differ from the previous frame are encoded to obtain encoded image data corresponding to each frame of the first image; and when the frame rate of the first image is greater than the preset frame rate, encode all regions of each frame of the first image to obtain encoded image data corresponding to each frame of the first image.

[0210] The embodiments provided in this application include image content type as a content parameter, which indicates the frame rate range of the first image. When the frame rate of the first image is less than or equal to a preset frame rate, the entire region of the first image in the first frame is encoded. Starting from the second image, only the regions in the first image that differ from the previous frame are encoded, resulting in encoded image data corresponding to each first image frame. When the frame rate of the first image is greater than the preset frame rate, the entire region of each first image frame is encoded, resulting in encoded image data corresponding to each first image frame. This achieves content-based classification encoding. Different compression qualities are selected for first images with different content types, allowing for selective encoding and transmission. This reduces the amount of data that needs to be transmitted, thereby saving available transmission bandwidth, reducing rendering pressure, and decreasing the frequency of adaptive rendering. This facilitates the use of efficient local rendering APIs to complete rendering, ensuring smooth screen projection.

[0211] In this embodiment, the content parameters include image content blocks, and the image content blocks include user-focused regions and non-focused regions; the first processing unit 502 is specifically used to: divide the user-focused regions and non-focused regions in the first image; encode the user-focused regions in each frame of the first image according to a first compression ratio and a first frame rate, and encode the non-focused regions in each frame of the first image according to a second compression ratio and a second frame rate, to obtain coded image data corresponding to each frame of the first image; wherein, the first compression ratio is less than the second compression ratio, and the first frame rate is greater than the second frame rate.

[0212] The embodiments provided in this application include image content segmentation, where each segment comprises user-focused and non-focused regions. The first image is divided into user-focused and non-focused regions. The user-focused regions in each frame of the first image are encoded according to a first compression ratio and a first frame rate, and the non-focused regions in each frame of the first image are encoded according to a second compression ratio and a second frame rate, resulting in encoded image data corresponding to each frame of the first image. The first compression ratio is less than the second compression ratio, and the first frame rate is greater than the second frame rate. This achieves spatially differentiated encoding based on image content segmentation, optimizes resource allocation, and saves bandwidth while maintaining low latency and high-quality projection between devices, enabling fast projection.

[0213] like Figure 10 As shown, this application embodiment provides a screen projection device 600, which is applied to a second device. The screen projection device 600 can be the second device or a component in the second device, such as an integrated circuit or a chip.

[0214] Specifically, the projection device 600 may include the second communication unit 602 and the second processing unit 604 described below.

[0215] The second communication unit 602 is used to receive data frames sent by the first device, wherein the data frames include at least the encoded image data of the first image;

[0216] The second processing unit 604 is used to acquire encoded image data and content parameters of the first image;

[0217] The second processing unit 604 is also used to decode and render the encoded image data using the decoding and rendering strategy corresponding to the content parameters to obtain the second image.

[0218] The screen projection device 600 provided in this application embodiment, for a second device, receives a data frame sent by a first device, the data frame including at least encoded image data of a first image; obtains the encoded image data and content parameters of the first image; and decodes and renders the encoded image data using a decoding and rendering strategy corresponding to the content parameters to obtain a second image. Through the aforementioned screen projection device 600, during the screen projection process, on the receiving end, based on the content parameters of the first image to be projected, a corresponding decoding and rendering strategy is selected for decoding and rendering operations. This achieves adaptive selection of decoding and rendering strategies based on content parameters, optimizes rendering resource allocation, improves the dynamic adaptive capability of screen projection between devices, and can adapt to the differences between different hardware platforms, thus balancing high performance, low latency, high quality, and universality of screen projection between devices.

[0219] In this embodiment, the content parameters include the image content type. The second processing unit 604 is specifically used to: determine candidate strategies for decoding and rendering the encoded image data based on the image content type and the operating system type of the second device; determine a decoding and rendering strategy from the list of available strategies supported by the second device based on the candidate strategies; and decode and render the encoded image data according to the decoding and rendering strategy to obtain the second image.

[0220] The embodiments provided in this application include content parameters such as image content type. Based on the image content type and the operating system type of the second device, candidate strategies for decoding and rendering the encoded image data are determined. Based on the candidate strategies, a decoding and rendering strategy is determined from the list of available strategies supported by the second device. The encoded image data is then decoded and rendered according to the decoding and rendering strategy to obtain a second image. This achieves a dynamic dual-decision logic of content-driven and resource-adaptive approaches. Facing diverse graphics cards and driver configurations, it can adaptively select decoding and rendering strategies, thereby solving the performance problems caused by general-purpose cross-platform rendering technologies. It has strong adaptability and can leverage the unique rendering advantages of different platforms.

[0221] In this embodiment of the application, the second processing unit 604 is further configured to: start a detection service program, detect the operability of the hardware decoder and different rendering application interfaces, and output the detection results; and determine a list of available strategies based on the detection results.

[0222] The embodiments provided in this application initiate a detection service program to detect the operability of the hardware decoder and different rendering application programming interfaces, and output the detection results; based on the detection results, a list of available strategies is determined. This facilitates broad adaptability of screen projection across platforms and hardware. Facing diverse graphics cards and driver configurations, it can adaptively select decoding and rendering strategies, thereby solving the performance problems caused by general-purpose cross-platform rendering technologies. It has strong adaptability and can leverage the unique rendering advantages of different platforms.

[0223] In this embodiment of the application, the content parameters include image content blocks, and the image content blocks include user-focused areas and non-focused areas. The second processing unit 604 is specifically used to: decode and render the encoded image data corresponding to the user-focused areas according to the first rendering strategy, and decode and render the encoded image data corresponding to the non-focused areas according to the second rendering strategy to obtain a second image; wherein the delay of the first rendering strategy is less than the delay of the second rendering strategy.

[0224] The embodiments provided in this application include image content blocks as content parameters. Each image content block includes user-focused and non-focused regions. The encoded image data corresponding to the user-focused regions is decoded and rendered according to a first rendering strategy, and the encoded image data corresponding to the non-focused regions is decoded and rendered according to a second rendering strategy to obtain a second image. The latency of the first rendering strategy is less than the latency of the second rendering strategy. This achieves spatially differentiated rendering based on image content blocks, optimizes resource allocation, prioritizes system resources for processing user-focused regions, and improves resource utilization efficiency. It can save bandwidth and improve the real-time performance of inter-device projection while maintaining low latency and high-quality effects. Furthermore, it represents a revolution in inter-device projection from content attributes to visual attention, demonstrating significant practical value in fields sensitive to real-time performance and visual quality, such as remote collaboration, cloud gaming, and in-vehicle projection.

[0225] In this embodiment, the second processing unit 604 is further configured to: acquire user eye-tracking information, interaction information, and screen content information of the projection interface; determine the user's gaze focus coordinates on the projection interface based on the eye-tracking information; determine the user's interaction focus area on the projection interface based on the interaction information; determine the semantic attention area of ​​the projection interface based on the screen content information; dynamically fuse and weight different areas of the projection interface based on the gaze focus coordinates, interaction focus area, semantic attention area, and time information; determine at least one user attention area based on the weight of different areas in the projection interface, and determine areas in the projection interface other than the user attention area as non-attention areas.

[0226] The embodiments provided in this application acquire user eye-tracking information, interaction information, and screen content information of the projection interface; determine the user's gaze focus coordinates on the projection interface based on the eye-tracking information; determine the user's interaction focus area on the projection interface based on the interaction information; determine the semantic attention area of ​​the projection interface based on the screen content information; dynamically fuse and weight different areas of the projection interface based on the gaze focus coordinates, interaction focus area, semantic attention area, and time information; determine at least one user attention area based on the weights of different areas in the projection interface, and determine the areas of the projection interface other than the user attention area as non-attention areas. Thus, by using parallel acquisition and fusion of multimodal signals, and dynamic decision-making based on context and weights, the user attention area is determined, improving the accuracy of image content segmentation.

[0227] The screen projection device 500 and screen projection device 600 in this application embodiment can be electronic devices or components within electronic devices, such as integrated circuits or chips. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the scope of the electronic device.

[0228] The screen projection device 500 and screen projection device 600 in this application embodiment can be devices with an operating system. The operating system can be Android, iOS, or other possible operating systems, and this application embodiment does not specifically limit them.

[0229] The screen projection device 500 provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method embodiments of this application, and the screen projection device 600 provided in this application embodiment can achieve Figure 3 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0230] Optionally, such as Figure 11 As shown, this application embodiment also provides an electronic device 700, including a processor 702 and a memory 704. The memory 704 stores a program or instructions that can run on the processor 702. When the program or instructions are executed by the processor 702, they implement the various steps of the above-described screen projection method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0231] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0232] Figure 12 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0233] Electronic device 800 includes, but is not limited to: radio frequency unit 801, network module 802, audio output unit 803, input unit 804, sensor 805, display unit 806, user input unit 807, interface unit 808, memory 809, and processor 810, etc.

[0234] Those skilled in the art will understand that the electronic device 800 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 810 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 12 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0235] When electronic device 800 is the first device:

[0236] Processor 810 is used to acquire the first image and the content parameters of the first image.

[0237] The processor 810 is also used to encode the first image according to the encoding strategy corresponding to the content parameters to obtain encoded image data.

[0238] Network module 802 is used to send data frames to a second device, the data frames including at least encoded image data.

[0239] In this embodiment, for the first device, a first image and its content parameters are acquired; the first image is encoded according to the encoding strategy corresponding to the content parameters to obtain encoded image data; a data frame is sent to the second device, so that the second device decodes and renders the encoded image data according to the decoding and rendering strategy corresponding to the content parameters to obtain the second image. In this embodiment, during the screen projection process, at the projection end, a corresponding encoding strategy is selected to encode the first image based on its content parameters. This achieves adaptive selection of the encoding strategy based on the content parameters, optimizes the allocation of encoding resources, and improves the dynamic adaptive capability of screen projection between devices.

[0240] Optionally, the content parameters include image content type, which is used to indicate the frame rate range of the first image. The processor 810 is specifically used to: encode all regions of the first image in the first frame when the frame rate of the first image is less than or equal to a preset frame rate; starting from the second image, only the regions in the first image that differ from the previous frame are encoded to obtain the encoded image data corresponding to each first image; and when the frame rate of the first image is greater than the preset frame rate, encode all regions of the first image in each frame to obtain the encoded image data corresponding to each first image.

[0241] The embodiments provided in this application include image content type as a content parameter, which indicates the frame rate range of the first image. When the frame rate of the first image is less than or equal to a preset frame rate, the entire region of the first image in the first frame is encoded. Starting from the second image, only the regions in the first image that differ from the previous frame are encoded, resulting in encoded image data corresponding to each first image frame. When the frame rate of the first image is greater than the preset frame rate, the entire region of each first image frame is encoded, resulting in encoded image data corresponding to each first image frame. This achieves content-based classification encoding. Different compression qualities are selected for first images with different content types, allowing for selective encoding and transmission. This reduces the amount of data that needs to be transmitted, thereby saving available transmission bandwidth, reducing rendering pressure, and decreasing the frequency of adaptive rendering. This facilitates the use of efficient local rendering APIs to complete rendering, ensuring smooth screen projection.

[0242] Optionally, the content parameters include image content blocks, which include user-focused and non-focused regions; the processor 810 is specifically used to: divide the user-focused and non-focused regions in the first image; encode the user-focused regions in each frame of the first image according to a first compression ratio and a first frame rate, and encode the non-focused regions in each frame of the first image according to a second compression ratio and a second frame rate, to obtain coded image data corresponding to each frame of the first image; wherein the first compression ratio is less than the second compression ratio, and the first frame rate is greater than the second frame rate.

[0243] The embodiments provided in this application include image content segmentation, where each segment comprises user-focused and non-focused regions. The first image is divided into user-focused and non-focused regions. The user-focused regions in each frame of the first image are encoded according to a first compression ratio and a first frame rate, and the non-focused regions in each frame of the first image are encoded according to a second compression ratio and a second frame rate, resulting in encoded image data corresponding to each frame of the first image. The first compression ratio is less than the second compression ratio, and the first frame rate is greater than the second frame rate. This achieves spatially differentiated encoding based on image content segmentation, optimizes resource allocation, and saves bandwidth while maintaining low latency and high-quality projection between devices, enabling fast projection.

[0244] When electronic device 800 is used as a second device:

[0245] Network module 802 is used to receive data frames sent by the first device, wherein the data frames include at least the encoded image data of the first image.

[0246] Processor 810 is used to acquire encoded image data and content parameters of the first image.

[0247] The processor 810 is also used to decode and render the encoded image data using a decoding and rendering strategy corresponding to the content parameters to obtain a second image.

[0248] In this embodiment, the second device receives a data frame sent by the first device, the data frame including at least the encoded image data of the first image; obtains the encoded image data and the content parameters of the first image; and decodes and renders the encoded image data using a decoding and rendering strategy corresponding to the content parameters to obtain the second image. In this embodiment, during the projection process, on the projected end, a corresponding decoding and rendering strategy is selected based on the content parameters of the first image to be projected for decoding and rendering operations. This achieves adaptive selection of the decoding and rendering strategy based on the content parameters, optimizes the allocation of rendering resources, improves the dynamic adaptive capability of projection between devices, and can adapt to the differences between different hardware platforms, thus balancing high performance, low latency, high quality, and universality of projection between devices.

[0249] Optionally, the content parameters include the image content type, and the processor 810 is specifically used to: determine candidate strategies for decoding and rendering the encoded image data based on the image content type and the operating system type of the second device; determine a decoding and rendering strategy from the list of available strategies supported by the second device based on the candidate strategies; and decode and render the encoded image data according to the decoding and rendering strategy to obtain the second image.

[0250] The embodiments provided in this application include content parameters such as image content type. Based on the image content type and the operating system type of the second device, candidate strategies for decoding and rendering the encoded image data are determined. Based on the candidate strategies, a decoding and rendering strategy is determined from the list of available strategies supported by the second device. The encoded image data is then decoded and rendered according to the decoding and rendering strategy to obtain a second image. This achieves a dynamic dual-decision logic of content-driven and resource-adaptive approaches. Facing diverse graphics cards and driver configurations, it can adaptively select decoding and rendering strategies, thereby solving the performance problems caused by general-purpose cross-platform rendering technologies. It has strong adaptability and can leverage the unique rendering advantages of different platforms.

[0251] Optionally, the processor 810 is also used to: start a detection service program, detect the operability of hardware decoders and different rendering application interfaces, and output the detection results; and determine a list of available strategies based on the detection results.

[0252] The embodiments provided in this application initiate a detection service program to detect the operability of the hardware decoder and different rendering application programming interfaces, and output the detection results; based on the detection results, a list of available strategies is determined. This facilitates broad adaptability of screen projection across platforms and hardware. Facing diverse graphics cards and driver configurations, it can adaptively select decoding and rendering strategies, thereby solving the performance problems caused by general-purpose cross-platform rendering technologies. It has strong adaptability and can leverage the unique rendering advantages of different platforms.

[0253] Optionally, the content parameters include image content blocks, which include user-focused regions and non-focused regions. The processor 810 is specifically used to: decode and render the encoded image data corresponding to the user-focused regions according to a first rendering strategy, and decode and render the encoded image data corresponding to the non-focused regions according to a second rendering strategy to obtain a second image; wherein the latency of the first rendering strategy is less than the latency of the second rendering strategy.

[0254] The embodiments provided in this application include image content blocks as content parameters. Each image content block includes user-focused and non-focused regions. The encoded image data corresponding to the user-focused regions is decoded and rendered according to a first rendering strategy, and the encoded image data corresponding to the non-focused regions is decoded and rendered according to a second rendering strategy to obtain a second image. The latency of the first rendering strategy is less than the latency of the second rendering strategy. This achieves spatially differentiated rendering based on image content blocks, optimizes resource allocation, prioritizes system resources for processing user-focused regions, and improves resource utilization efficiency. It can save bandwidth and improve the real-time performance of inter-device projection while maintaining low latency and high-quality effects. Furthermore, it represents a revolution in inter-device projection from content attributes to visual attention, demonstrating significant practical value in fields sensitive to real-time performance and visual quality, such as remote collaboration, cloud gaming, and in-vehicle projection.

[0255] Optionally, the processor 810 is further configured to: acquire user eye-tracking information, interaction information, and screen content information of the projection interface; determine the coordinates of the user's gaze focus on the projection interface based on the eye-tracking information; determine the user's interaction focus area on the projection interface based on the interaction information; determine the semantic attention area of ​​the projection interface based on the screen content information; dynamically fuse and weight different areas of the projection interface based on the gaze focus coordinates, interaction focus area, semantic attention area, and time information; determine at least one user attention area based on the weights of different areas in the projection interface, and define areas in the projection interface other than the user attention area as non-attention areas.

[0256] The embodiments provided in this application acquire user eye-tracking information, interaction information, and screen content information of the projection interface; determine the user's gaze focus coordinates on the projection interface based on the eye-tracking information; determine the user's interaction focus area on the projection interface based on the interaction information; determine the semantic attention area of ​​the projection interface based on the screen content information; dynamically fuse and weight different areas of the projection interface based on the gaze focus coordinates, interaction focus area, semantic attention area, and time information; determine at least one user attention area based on the weights of different areas in the projection interface, and determine the areas of the projection interface other than the user attention area as non-attention areas. Thus, by using parallel acquisition and fusion of multimodal signals, and dynamic decision-making based on context and weights, the user attention area is determined, improving the accuracy of image content segmentation.

[0257] It should be understood that, in this embodiment, the input unit 804 may include a graphics processing unit (GPU) 8041 and a microphone 8042. The GPU 8041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 806 may include a display panel 8061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 may include a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0258] The memory 809 can be used to store software programs and various data. The memory 809 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 809 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 809 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0259] Processor 810 may include one or more processing units; optionally, processor 810 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 810.

[0260] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described screen projection method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0261] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0262] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described screen projection method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0263] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0264] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the screen projection method embodiment described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0265] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0266] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0267] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A screen projection method, characterized by, The screen projection method, executed by the first device, includes: Obtain the first image and its content parameters; According to the encoding strategy corresponding to the content parameters, the first image is encoded to obtain encoded image data. A data frame is sent to a second device, the data frame including at least the encoded image data.

2. The screen projection method of claim 1, wherein, The content parameters include image content type, which indicates the frame rate range of the first image. The encoding process, based on the encoding strategy corresponding to the content parameters, performs encoding processing on the first image to obtain encoded image data, including: When the frame rate of the first image is less than or equal to the preset frame rate, the entire region of the first image in the first frame is encoded. Starting from the first image in the second frame, only the regions in the first image that differ from the previous frame are encoded to obtain the encoded image data corresponding to the first image in each frame. When the frame rate of the first image is greater than the preset frame rate, the entire region of the first image in each frame is encoded to obtain the encoded image data corresponding to the first image in each frame.

3. The screen projection method of claim 1, wherein, The content parameters include image content blocks, which include user-focused and non-focused regions; the encoding process of the first image according to the encoding strategy corresponding to the content parameters to obtain encoded image data includes: Divide the first image into user-focused and non-focused regions; The user-interested regions in each frame of the first image are encoded according to a first compression ratio and a first frame rate, and the non-interested regions in each frame of the first image are encoded according to a second compression ratio and a second frame rate, to obtain the encoded image data corresponding to each frame of the first image. Wherein, the first compression ratio is less than the second compression ratio, and the first frame rate is greater than the second frame rate.

4. A screen projection method, characterized by, The screen projection method, executed by a second device, includes: Receive a data frame sent by a first device, the data frame including at least the encoded image data of a first image; Obtain the encoded image data and the content parameters of the first image; The encoded image data is decoded and rendered using the decoding and rendering strategy corresponding to the content parameters to obtain the second image.

5. The screen projection method of claim 4, wherein, The content parameters include image content type. The step of decoding and rendering the encoded image data using the decoding and rendering strategy corresponding to the content parameters to obtain the second image includes: Based on the image content type and the operating system type of the second device, candidate strategies for decoding and rendering the encoded image data are determined; Based on the candidate strategies, the decoding and rendering strategy is determined from the list of available strategies supported by the second device; The encoded image data is decoded and rendered according to the decoding and rendering strategy to obtain the second image.

6. The screen projection method of claim 4, wherein, The content parameters include image content blocks, which include user-focused and non-focused regions. The process of decoding and rendering the encoded image data using the decoding and rendering strategy corresponding to the content parameters to obtain a second image includes: The encoded image data corresponding to the user's area of ​​interest is decoded and rendered according to the first rendering strategy, and the encoded image data corresponding to the non-area of ​​interest is decoded and rendered according to the second rendering strategy to obtain the second image; The latency of the first rendering strategy is less than the latency of the second rendering strategy.

7. The screen projection method of claim 6, wherein, The screen mirroring method also includes: Acquire user eye-tracking information, interaction information, and screen content information of the projection interface; The coordinates of the user's gaze focus on the projection interface are determined based on the eye-tracking information. The user's focus area on the projection interface is determined based on the interaction information. The semantic attention area of ​​the projection interface is determined based on the content information of the screen. Based on the gaze focus coordinates, the interaction focus area, the semantic attention area, and time information, different areas of the projection interface are dynamically fused and weighted. At least one user-focused area is determined based on the weight of different areas in the projection interface, and the areas in the projection interface other than the user-focused area are determined as the non-focused areas.

8. A screen projection device, characterized by, Applied to the first device, the projection device includes: The first processing unit is configured to acquire the first image and the content parameters of the first image; The first processing unit is further configured to encode the first image according to the encoding strategy corresponding to the content parameters to obtain encoded image data; A first communication unit is configured to send a data frame to a second device, the data frame including at least the encoded image data.

9. The screen projection device according to claim 8, wherein, The content parameters include image content type, which indicates the frame rate range of the first image. The first processing unit is specifically used for: When the frame rate of the first image is less than or equal to the preset frame rate, the entire region of the first image in the first frame is encoded. Starting from the first image in the second frame, only the regions in the first image that differ from the previous frame are encoded to obtain the encoded image data corresponding to the first image in each frame. When the frame rate of the first image is greater than the preset frame rate, the entire region of the first image in each frame is encoded to obtain the encoded image data corresponding to the first image in each frame.

10. The screen projection device of claim 8, wherein, The content parameters include image content blocks, which include user-focused and non-focused regions; the first processing unit is specifically used for: Divide the first image into user-focused and non-focused regions; The user-interested regions in each frame of the first image are encoded according to a first compression ratio and a first frame rate, and the non-interested regions in each frame of the first image are encoded according to a second compression ratio and a second frame rate, to obtain the encoded image data corresponding to each frame of the first image. Wherein, the first compression ratio is less than the second compression ratio, and the first frame rate is greater than the second frame rate.

11. A screen projection device, characterized by, The projection device, used in a second device, includes: The second communication unit is used to receive data frames sent by the first device, wherein the data frames include at least the encoded image data of the first image; The second processing unit is used to acquire the encoded image data and the content parameters of the first image; The second processing unit is further configured to decode and render the encoded image data using the decoding and rendering strategy corresponding to the content parameters to obtain a second image.

12. The projection device according to claim 11, characterized in that, The content parameters include image content type, and the second processing unit is specifically used for: Based on the image content type and the operating system type of the second device, candidate strategies for decoding and rendering the encoded image data are determined; Based on the candidate strategies, the decoding and rendering strategy is determined from the list of available strategies supported by the second device; The encoded image data is decoded and rendered according to the decoding and rendering strategy to obtain the second image.

13. The projection device according to claim 11, characterized in that, The content parameters include image content blocks, which include user-focused and non-focused regions. The second processing unit is specifically used for: The encoded image data corresponding to the user's area of ​​interest is decoded and rendered according to the first rendering strategy, and the encoded image data corresponding to the non-area of ​​interest is decoded and rendered according to the second rendering strategy to obtain the second image; The latency of the first rendering strategy is less than the latency of the second rendering strategy.

14. The projection device according to claim 13, characterized in that, The second processing unit is further configured to: Acquire user eye-tracking information, interaction information, and screen content information of the projection interface; The coordinates of the user's gaze focus on the projection interface are determined based on the eye-tracking information. The user's focus area on the projection interface is determined based on the interaction information. The semantic attention area of ​​the projection interface is determined based on the content information of the screen. Based on the gaze focus coordinates, the interaction focus area, the semantic attention area, and time information, different areas of the projection interface are dynamically fused and weighted. At least one user-focused area is determined based on the weight of different areas in the projection interface, and the areas in the projection interface other than the user-focused area are determined as the non-focused areas.

15. An electronic device, characterized in that, It includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the screen projection method as described in any one of claims 1 to 3, and / or, the programs or instructions being executed by the processor to implement the steps of the screen projection method as described in any one of claims 4 to 7.

Images