Hybrid coding method and apparatus, and hybrid decoding method and apparatus

Abstract

Provided in the present application are a hybrid coding method and apparatus, and a hybrid decoding method and apparatus. The hybrid coding method of the present application comprises: acquiring content information of a region to be coded, the content information being used for indicating the content of said region; determining a processing mode of said region on the basis of the content information, the processing mode comprising a first processing mode or a second processing mode; and coding the content of said region on the basis of the processing mode of said region so as to obtain a code stream. The present application can reduce the coding code stream, reduce power consumption and complexity, and improve the screen projection effect.

Description

Hybrid encoding and decoding methods and apparatus Technical Field

[0001] This application relates to screen projection technology, and more particularly to a hybrid encoding and decoding method and apparatus. Background Technology

[0002] Screen mirroring refers to the process of transferring the screen display of one terminal device (e.g., mobile phone, personal computer (PC), tablet, TV, etc.) to the screen of another terminal device (e.g., mobile phone, PC, tablet, TV, etc.). Common screen mirroring applications include mirroring a mobile phone desktop to a TV, conference presentations, remote desktops, and collaborative work using PC / tablet desktops. In a screen mirroring application, the two terminal devices can be designated as the mirroring end or the receiving end. The mirroring end is the terminal device that initiates the mirroring, and the receiving end is the terminal device that receives and displays the mirrored content.

[0003] In related screen mirroring technologies, the screen mirroring device can encode a sequence of desktop images into a bitstream and send it to the receiving device, which then decodes, reconstructs, and displays the image. Alternatively, the screen mirroring device can encode rendering instructions and resources into a bitstream and send it to the receiving device, which then decodes, renders, and displays the image. However, both of these methods suffer from the problem that in some cases, the large amount of data can lead to an excessively high transmission bitrate, affecting the screen mirroring effect. Summary of the Invention

[0004] This application provides a hybrid encoding and decoding method and apparatus to reduce encoding bitrate, power consumption and complexity, and improve screen projection effect.

[0005] In a first aspect, this application provides a hybrid encoding method, comprising: acquiring content information of a region to be encoded, the content information being used to indicate the content of the region to be encoded; determining a processing mode for the region to be encoded based on the content information, the processing mode including a first processing mode or a second processing mode; and encoding the content of the region to be encoded according to the processing mode of the region to be encoded to obtain a bitstream.

[0006] In this embodiment, the processing mode applicable to the region to be encoded is determined based on the content information of the region to be encoded. This processing mode can provide greater benefits compared to another processing mode. On the one hand, it realizes the integration of the two processing modes, and on the other hand, it can encode the content of the region to be encoded in the processing mode most suitable for it, which can reduce the encoding bitrate, reduce power consumption and complexity, and improve the projection effect.

[0007] The area to be encoded can include the entire area of ​​the current desktop or the area corresponding to at least one application or window on the current desktop. The current desktop refers to the screen of the projection device at the current moment. Based on this, the entire area of ​​the current desktop can refer to the entire screen of the projection device; while the area corresponding to at least one application or window on the current desktop can refer to the area occupied by at least one application or window displayed on the current desktop on the screen of the projection device. This area can occupy the entire screen of the projection device (full-screen display) or a portion of the screen of the projection device (small window or split-screen display).

[0008] It should be understood that when the area to be encoded refers to the area corresponding to at least one application or window on the current desktop, the size of the area to be encoded depends on the size of the area occupied by at least one application or window. When at least one application or window occupies the entire screen, the area to be encoded is the entire area of ​​the current desktop. When at least one application or window occupies part of the screen, the area to be encoded is part of the current desktop. This application does not specifically limit the size of the area to be encoded in this case.

[0009] The content information of the region to be encoded is used to indicate the content of the region to be encoded. In this application, the content information may include the following two types:

[0010] (1) The content information includes first information and second information. The first information includes resource information of the content of the region to be encoded in the first processing mode, and the second information includes resource information of the content of the region to be encoded in the second processing mode.

[0011] In one possible implementation, in the first processing mode, the content of the region to be encoded includes images and / or video within the region to be encoded. Accordingly, the first information includes image information and / or video information, wherein the image information includes the number of images (N). img ) or image resolution (IW i ×IH i {i=1,2,3,…,N img The video information includes at least one of the following: (also known as image resolution), and the video information includes the number of videos (N). buf Since videos are typically stored in a buffer in a queue, this is also referred to as the number of buffer queues, and the video resolution (BW). j ×BH j {j=1,2,3,…,N buf}), the display resolution corresponding to the video (DW) k ×DH k {k = 1, 2, 3, ..., N} buf}) or the corresponding encoding frame rate of the video (FR) m {m=1,2,3,…,N bufAt least one of}).

[0012] In one possible implementation, in the second processing mode, the content of the region to be encoded includes a sequence of desktop images of the region to be encoded, and correspondingly, the second information includes the resolution (W) of the desktop images in the desktop image sequence. s ×H s ), the refresh rate of the desktop image sequence (F) ps ) or the encoded frame rate (F) of a desktop image sequence r At least one of them.

[0013] Optionally, the first processing mode can be a processing mode based on the rendered data.

[0014] Optionally, the second processing mode can be a desktop image sequence-based processing mode.

[0015] It should be noted that the above exemplary description of the data contained in the first information and the second information does not constitute a limitation. The data contained in the first information and the second information mainly depend on the resources required by the corresponding processing mode. Therefore, this application does not specifically limit the first information and the second information.

[0016] (2) The content information includes third information, which includes the attribute information of the content of the area to be encoded, including at least one of the content identifier, content name, or content display level.

[0017] The third piece of information includes, for example, the name, identifier, and display hierarchy of the application / system window program. For instance, as shown in Figure 4, the third piece of information includes the name of the video application displayed at the top of the current desktop and the display hierarchy of that video application.

[0018] It should be noted that the above examples describe the data contained in the third information, but this does not constitute a limitation. The data contained in the third information mainly depends on the rendering object (application / system window program), so this application does not make specific limitations on the third information.

[0019] In this application, the content information can be obtained from the rendering information of the application or system window program being rendered, which is received in the rendering service (e.g., the system's rendering service or the application's rendering service), or it can be obtained by reading the current system task manager, without any specific limitation.

[0020] The projection device can decide the processing mode of the area to be encoded in the following two ways:

[0021] (1) Compare the benefits of the first processing mode and the second processing mode based on the first information and the second information; determine the processing mode of the region to be encoded as the one with the greater benefit between the first processing mode and the second processing mode.

[0022] Benefits can refer to the advantageous effects obtained after processing by the first or second processing mode. Therefore, significant benefits can include, for example, low encoding bitrate, low power consumption, and low complexity. In other words, the mode with the lower encoding bitrate, lower power consumption, or lower complexity among the first and second processing modes can be determined as the processing mode for the region to be encoded.

[0023] In one possible implementation, the benefit of the second processing mode is determined to be greater than the benefit of the first processing mode when at least one of the following first conditions is met; the benefit of the first processing mode is determined to be greater than the benefit of the second processing mode when none of the first conditions are met. The first conditions include the following:

[0024] ① When the number of videos is greater than 0 (N) buf When W > 0), the resolution of the desktop image is less than or equal to the sum of the resolutions of multiple videos (which can be videos in the buffer queue); s ×H s ≤∑ j (BW j ×BH j )

[0025] j≥1.

[0026] When this condition is met, the second processing mode is expected to save more bitrate than the first processing mode, so the second processing mode is more profitable.

[0027] ② When the number of videos is greater than 0 (N) buf When the value is greater than 0, the sum of the resolutions of multiple videos (which may be videos in the cache queue) is less than the resolution of the desktop image, and the sum of the resolutions of the multiple videos is greater than the product of the sum of the display resolutions corresponding to the multiple videos and the first preset value; W s ×H s >Σ j (BW j ×BH j )>∑ k (DW k ×DH k )×R0

[0028] j≥1; k≥1; R0 is a predefined constant, optional, 1≤R0≤2.

[0029] When this condition is met, the second processing mode is expected to save more bitrate than the first processing mode, so the second processing mode is more profitable.

[0030] ③ The number of videos is greater than the second preset value (N) buf>N0), the second preset value is a positive integer (N0≥1);

[0031] When this condition is met, the first processing mode requires encoding multiple videos, thus necessitating the execution of multiple video encoders. This is anticipated to exceed the real-time encoding capabilities of the video encoders, increasing the system's processing burden. In contrast, the second processing mode only requires a single video encoder, resulting in lower system complexity and lower power consumption; therefore, the second processing mode offers greater benefits.

[0032] ④ When the number of videos is greater than 0 (N) buf When the resolution of multiple images is greater than 0, the sum of their resolutions is greater than the sum of the resolutions of the desktop images refreshed within a preset time period (e.g., 1 / F0 seconds). s ×H s ×F ps / F0≤∑ i (IW i ×IH i )

[0033] i≥1; F0 is a positive integer constant, optional, 0 <F0≤10。

[0034] When this condition is met, the first processing mode may generate a large instantaneous bitrate. Using the second processing mode is expected to save bitrate and provide a smoother, more lag-free display. Therefore, the second processing mode is more beneficial.

[0035] ⑤ When the number of videos is greater than 0 (N) buf When the sum of the encoded frame rates of the video is greater than the refresh rate of the desktop image, or the sum of the encoded frame rates of the video is greater than the encoded frame rate of the desktop image sequence. ps <∑ m FR m

[0036] or,

[0037] F r <∑ m FR m

[0038] m≥1.

[0039] When this condition is met, the first processing mode may generate greater power consumption and bit rate. Using the second processing mode is expected to result in lower power consumption and lower bit rate, thus the second processing mode is more beneficial.

[0040] Based on this, when one of the above five conditions is met, the processing mode for the region to be encoded can be determined as the second processing mode. Conversely, when none of the above five conditions are met, the processing mode for the region to be encoded can be determined as the first processing mode.

[0041] Optionally, when the number of videos (N) buf When =0), the processing mode of the region to be encoded can be decided as the first processing mode.

[0042] Optionally, when there are no rendering resources (i.e., corresponding to the number N images in the content information), img =0 and the number of videos N buf When =0), the processing mode of the region to be encoded can be decided as the first processing mode.

[0043] (2) Based on the pre-set correspondence, the processing mode corresponding to the third information is determined as the processing mode of the region to be encoded.

[0044] The third piece of information can indicate the application / system window program currently on the desktop. Based on this, the application scenario of the current desktop can be inferred, and the processing mode of the area to be coded can be directly determined. For example, if the current desktop displays a browser-type application, it can be inferred that it contains a lot of text and images, so the first processing mode can be used; or, if the current desktop displays an office software-type application, it can be inferred that it mainly contains text and images and no video, so the first processing mode can be used; or, if the current desktop only contains the system interface and does not display any applications, the first processing mode can be used; or, if the current desktop displays a video playback-type application, it can be inferred that it contains video playback, so the second processing mode can be used.

[0045] It should be noted that the decision-making methods of the two processing modes mentioned above are only examples. In specific implementation, this application may also consider other decision-making methods to determine the processing mode of the region to be encoded as the one with greater benefit between the first processing mode and the second processing mode. Therefore, no specific limitation is made on the decision-making method.

[0046] Based on the decision results regarding the processing mode of the region to be encoded, the projection terminal can include the following situations when encoding the content of the region to be encoded:

[0047] 1. The processing mode for the region to be encoded is the first processing mode.

[0048] 1.1 The projection terminal obtains the rendering instruction, which corresponds to the content of the area to be encoded; the rendering instruction is encoded to obtain the bitstream.

[0049] Rendering commands can originate from applications, system windows, etc., on the current desktop of the projection device. The projection device can capture user operation commands from the screen or other input modules, then generate rendering commands in response to these operation commands and send them to the rendering service. In this way, rendering commands can be obtained from the rendering service.

[0050] In this application, the rendering service intercepts rendering instructions generated by all applications and system window programs running in the system, and can then render the image of the current desktop based on these rendering instructions.

[0051] 1.2 In addition to encoding rendering instructions, the projection terminal can also encode rendering resources, that is, obtain the resources corresponding to the content of the area to be encoded in the first processing mode according to the rendering instructions. The resources include at least one image and / or at least one video. The resources corresponding to the content of the area to be encoded in the first processing mode are encoded to obtain the bitstream.

[0052] The resources corresponding to the content of the region to be encoded (also known as rendering resources) can be obtained based on the parsing of rendering instructions. Rendering resources contain at least one image and / or at least one video (video can be stored in a buffer as a queue). Rendering resources occupy a large amount of storage space, and applications or system window programs typically include these resources in the rendering instructions with the most economical signal data (e.g., index numbers) and send them to the rendering service. During the rendering process, the rendering service parses the rendering instructions to obtain the aforementioned signal data, and then locates these rendering resources in the system path, application, or system window program based on this signal data before performing the rendering.

[0053] In this application, in addition to the above-mentioned process of obtaining rendering resources, if the application or system window program directly sends the required rendering resources to the rendering service, the projection terminal can also directly obtain the rendering resources from the rendering service.

[0054] In the first processing mode, rendering instructions need to be encoded. If necessary, resources (including images and / or videos) corresponding to the content of the area to be encoded also need to be encoded. Rendering instructions can be encoded using data encoding methods, images can be encoded using image encoding methods, and videos can be encoded using a video encoder. This application does not limit the specific implementation of the aforementioned encoding methods.

[0055] 2. The processing mode for the region to be encoded is the second processing mode.

[0056] 2.1 The projection terminal acquires the resources corresponding to the content of the area to be encoded in the second processing mode, which includes a desktop image sequence; the resources corresponding to the content of the area to be encoded in the second processing mode are encoded to obtain a bitstream.

[0057] In this application, the resources corresponding to the content of the region to be encoded can be obtained by rendering the rendering data once, or by accessing the memory, without any specific limitation.

[0058] Accordingly, the resolution of the desktop images in the desktop image sequence included in the second information above can be obtained by parsing the rendering instructions or by obtaining them from the system, without any specific limitation; the refresh rate or the encoding frame rate of the desktop image sequence can be obtained from the rendering service or by obtaining them from the system, without any specific limitation.

[0059] 2.2 In addition to encoding the resources corresponding to the content of the area to be encoded, the projection terminal also encodes rendering instructions, that is, it obtains rendering instructions, which correspond to the content of the area to be encoded; and encodes the rendering instructions to obtain the bitstream.

[0060] In the second processing mode, resources corresponding to the content of the area to be encoded (including desktop image sequences) need to be encoded. If necessary, rendering instructions also need to be encoded (to ensure the continuity of rendering instructions; if not continuous, more rendering data needs to be encoded and sent when switching from the second processing mode to the first processing mode in order for the image to be correctly rendered at the receiving end). The desktop image sequence can be encoded using a video encoder, and the rendering instructions can be encoded using data encoding. This application does not limit the specific implementation of the aforementioned encoding methods. It should be noted that in the second processing mode, the projection end may not encode the rendering instructions, but only encode the resources corresponding to the content of the area to be encoded (including desktop image sequences). In this case, when switching from the second processing mode to the first processing mode, the projection end may need to encode more rendering instruction information so that the receiving end can correctly parse the rendering instructions.

[0061] Optionally, the bitstream may also include processing mode information, which indicates the processing mode of the region to be encoded.

[0062] In this way, the receiving end can obtain the processing mode of the region to be decoded by parsing the bitstream. This processing mode can be either the first processing mode or the second processing mode.

[0063] In one possible implementation, the projection end and the receiving end can inform the receiving end of the processing mode of the area to be decoded through an agreed-upon method. For example, the projection end places the bitstream of the rendering instructions in a frame at the end of the frame's bitstream (for example, a frame's bitstream of the area to be encoded may include the bitstream of the rendering instructions and the bitstream of the resources corresponding to the content of the area to be encoded; in this case, the bitstream of the resources corresponding to the content of the area to be encoded is placed first, followed by the bitstream of the rendering instructions; if there are no other bitstreams, then the bitstream of the rendering instructions is at the end of the bitstream of the area to be encoded in that frame). When the receiving end parses the bitstream of the rendering instructions in a frame, it determines whether the bitstream of that frame contains the bitstream of an image / video. If not, the processing mode of the area to be decoded in that frame is the first processing mode; if it does, it further determines whether the resolution of the image / video is the same as the resolution of the desktop image of the area to be decoded (to achieve this determination, the projection end can write the above content information into the bitstream). If they are the same, the processing mode of the area to be decoded in that frame is the second processing mode; if they are different, the processing mode of the area to be decoded in that frame is the first processing mode. It should be noted that, in addition to the above examples, the projection end and the receiving end may also use other agreed-upon methods to inform the receiving end of the processing mode of the area to be decoded, and this application does not impose specific limitations on this.

[0064] Optionally, the bitstream may also include first data type information, which is used to indicate the bitstream corresponding to the rendering instructions.

[0065] Optionally, the bitstream may also include second data type information and / or third data type information, wherein the second data type information is used to indicate the bitstream corresponding to at least one image, and the third data type information is used to indicate the bitstream corresponding to at least one video.

[0066] Optionally, the bitstream may also include fourth data type information, which is used to indicate the bitstream corresponding to the desktop image sequence.

[0067] In one possible implementation, based on at least one of the aforementioned first, second, third, and fourth data types, the projection end and the receiving end can synchronize the processing mode of the area to be encoded / decoded. That is, the encoding end carries the data type information of the corresponding bitstream in the bitstream; the receiving end decompresses the bitstream, and when it obtains the fourth data type information, it can determine that the processing mode of the area to be encoded / decoded is the second processing mode; or, when it obtains the second and / or third data type information, it can determine that the processing mode of the area to be encoded / decoded is the first processing mode.

[0068] The projection device can directly write the data type corresponding to the bitstream into the bitstream, such as the bitstream of rendering instructions, the bitstream of images, the bitstream of videos, or the bitstream of desktop image sequences, so that the receiving device can parse the bitstream based on the corresponding decoding method.

[0069] In this way, the receiving end can determine the type of the bitstream by parsing the data type information in the bitstream of the area to be decoded, and then use the decoder corresponding to the projection end to decode the different types of bitstreams and reconstruct them into rendering instructions (the bitstream includes the first data type information), images (the bitstream includes the second data type information), videos (the bitstream includes the third data type information), desktop image sequences (the bitstream includes the fourth data type information), etc.

[0070] Secondly, this application provides a hybrid decoding method, comprising: receiving a bitstream; determining a processing mode for a region to be decoded based on the bitstream, the processing mode including a first processing mode or a second processing mode; obtaining data corresponding to the content of the region to be decoded based on the bitstream; and obtaining a screen to be displayed based on the processing mode of the region to be decoded and the data corresponding to the content of the region to be decoded.

[0071] In this embodiment, the processing mode suitable for the area to be decoded is determined based on the bitstream transmitted by the projection end. This processing mode can provide greater benefits compared to another processing mode. On the one hand, it realizes the integration of the two processing modes, and on the other hand, it can render the content of the area to be decoded in the processing mode most suitable for it, which can improve transmission efficiency, reduce power consumption and complexity, and improve the projection effect.

[0072] The processing mode includes a first processing mode or a second processing mode. Optionally, the first processing mode can be the processing mode based on rendered data as described above. Optionally, the second processing mode can be the processing mode based on desktop image sequences as described above.

[0073] The receiving end can decide the processing mode of the region to be decoded in the following two ways:

[0074] (1) The bitstream includes processing mode information, and the processing mode of the region to be decoded is determined based on the processing mode information.

[0075] The receiving end can parse the bitstream to obtain processing mode information, and then determine the processing mode of the region to be decoded based on the processing mode information.

[0076] (2) Obtain the content information and data type information of the region to be decoded based on the bitstream. The content information of the region to be decoded is used to indicate the content of the region to be decoded, and the data type information is used to indicate the bitstream of the rendering instruction, the bitstream of the image, the bitstream of the video, or the bitstream of the desktop image sequence. Determine the processing mode of the region to be decoded based on the content information and data type information of the region to be decoded.

[0077] The receiving end can determine the processing mode of the region to be decoded using the aforementioned method. For example, when the receiving end parses the bitstream of rendering instructions in a frame, it determines whether the bitstream of that frame contains an image / video bitstream. If not, the processing mode of the region to be decoded in that frame is the first processing mode; if it does, it further determines whether the resolution of the image / video is the same as the resolution of the desktop image in the region to be decoded (to achieve this determination, the projection end can write content information into the bitstream). If they are the same, the processing mode of the region to be decoded in that frame is the second processing mode; if they are different, the processing mode of the region to be decoded in that frame is the first processing mode. It should be noted that, in addition to the above example, the projection end and the receiving end can also use other agreed-upon methods to inform the receiving end of the processing mode of the region to be decoded, and this application does not specifically limit this.

[0078] Optionally, the bitstream may also include first data type information, which is used to indicate the bitstream corresponding to the rendering instructions.

[0079] Optionally, the bitstream may also include second data type information and / or third data type information, wherein the second data type information is used to indicate the bitstream corresponding to at least one image, and the third data type information is used to indicate the bitstream corresponding to at least one video.

[0080] Optionally, the bitstream may also include fourth data type information, which is used to indicate the bitstream corresponding to the desktop image sequence.

[0081] The receiving end can obtain one or more of the above-mentioned data types based on the bitstream, thereby determining the data type corresponding to one or more segments of the bitstream, such as the bitstream of rendering instructions, the bitstream of images, the bitstream of videos, or the bitstream of desktop image sequences, and then parse the bitstream based on the corresponding decoding method.

[0082] In the first processing mode, the projection end can encode rendering instructions, and if necessary, can also encode resources (including images and / or videos) corresponding to the content of the area to be encoded. Rendering instructions can be encoded using data encoding methods, images using image encoding methods, and videos using video encoders. Correspondingly, the receiving end can decode the rendering instruction stream using data decoding methods, decode the image stream using image decoding methods, and decode the video stream using video decoding methods. In this mode, the data corresponding to the content of the area to be decoded includes rendering instructions, and may also include images and / or videos (also known as rendering resources).

[0083] In the second processing mode, the projection end can encode resources (including desktop image sequences) corresponding to the content of the area to be encoded. If necessary, it can also encode rendering instructions. The desktop image sequence can be encoded using a video encoder, and the rendering instructions can be encoded using data encoding. This application does not limit the specific implementation of the aforementioned encoding methods. Correspondingly, the receiving end can use video decoding to decode the bitstream of the desktop image sequence to obtain the desktop image sequence, and use data decoding to decode the bitstream of the rendering instructions to obtain the rendering instructions. In this case, the data corresponding to the content of the area to be decoded includes the desktop image sequence, and may also include rendering instructions.

[0084] Based on the processing mode of the area to be decoded determined by the above steps, the receiving end can obtain the screen to be displayed in the following ways:

[0085] 1. The processing mode for the area to be decoded is the first processing mode.

[0086] 1.1. Render the image to be displayed according to the rendering instructions.

[0087] In one possible implementation, under the first processing mode, the projection end may not encode the rendering resources. In this case, the receiving end can directly render according to the rendering instructions and draw the screen to be displayed.

[0088] In one possible implementation, in the first processing mode, the projection end may not encode the rendering resources. In this case, the receiving end can obtain signal data (such as index number) by parsing the rendering instructions, and then find the corresponding rendering resources in the system path, application, or system window program based on these signal data.

[0089] The receiving end renders the rendering resources obtained through the above processing according to the rendering instructions obtained from the decompressed bitstream, and can obtain the screen to be displayed.

[0090] 1.2. Render at least one image and / or at least one video according to the rendering instructions to obtain the screen to be displayed.

[0091] The receiving end obtains rendering instructions and rendering resources (including at least one image and / or at least one video) from the decompressed bitstream, and then renders the rendering resources based on the rendering instructions to obtain the screen to be displayed.

[0092] 2. The processing mode for the area to be decoded is the second processing mode.

[0093] 2.1 Use the desktop image sequence as the screen to be displayed.

[0094] After the receiving end decompresses the bitstream and reconstructs the desktop image sequence, it can be directly used as the screen to be displayed without rendering.

[0095] 2.2 Render the desktop image sequence according to the rendering instructions to obtain the screen to be displayed.

[0096] After the receiving end decompresses the bitstream and reconstructs the desktop image sequence, it can also render the image based on rendering instructions to obtain the image to be displayed, thereby improving the visual appeal of the image to be displayed.

[0097] Thirdly, this application provides a hybrid encoding device, comprising: an acquisition module for acquiring content information of a region to be encoded, the content information indicating the content of the region to be encoded; a decision module for deciding a processing mode of the region to be encoded based on the content information, the processing mode including a first processing mode or a second processing mode; and an encoding module for encoding the content of the region to be encoded according to the processing mode of the region to be encoded to obtain a bitstream.

[0098] In one possible implementation, the content information includes first information and second information, wherein the first information includes resource information of the content of the region to be encoded in the first processing mode, and the second information includes resource information of the content of the region to be encoded in the second processing mode.

[0099] In one possible implementation, the decision module is specifically configured to compare the benefits of the first processing mode and the second processing mode based on the first information and the second information; and determine the processing mode of the region to be encoded as the one with the greater benefit between the first processing mode and the second processing mode.

[0100] In one possible implementation, in the first processing mode, the content of the region to be encoded includes images and / or videos within the region to be encoded. Accordingly, the first information includes information about the images and / or the videos, wherein the image information includes at least one of the number of images or the resolution of the images, and the video information includes at least one of the number of videos, the resolution of the videos, the display resolution corresponding to the videos, or the encoding frame rate of the videos. In the second processing mode, the content of the region to be encoded includes a sequence of desktop images within the region to be encoded. Accordingly, the second information includes at least one of the resolution of the desktop images in the desktop image sequence, the refresh rate of the desktop image sequence, or the encoding frame rate of the desktop image sequence.

[0101] In one possible implementation, the decision module is specifically configured to determine that the benefit of the second processing mode is greater than the benefit of the first processing mode when at least one of the first conditions is met; and to determine that the benefit of the first processing mode is greater than the benefit of the second processing mode when none of the conditions in the first conditions are met; the first conditions include the following:

[0102] When the number of videos is greater than 0, the resolution of the desktop image is less than or equal to the sum of the resolutions of the multiple videos; or,

[0103] When the number of videos is greater than 0, the sum of the resolutions of the multiple videos is less than the resolution of the desktop image, and the sum of the resolutions of the multiple videos is greater than the product of the sum of the display resolutions corresponding to the multiple videos and a first preset value; or,

[0104] The number of videos is greater than a second preset value, where the second preset value is a positive integer; or...

[0105] When the number of videos is greater than 0, the sum of the resolutions of the multiple images is greater than the sum of the resolutions of the desktop images refreshed within a preset time period; or,

[0106] When the number of videos is greater than 0, the sum of the encoded frame rates of the multiple videos is greater than the refresh rate of the desktop image sequence; or,

[0107] When the number of videos is greater than 0, the sum of the encoded frame rates of the multiple videos is greater than the encoded frame rate of the desktop image sequence.

[0108] In one possible implementation, the content information includes third information, which includes attribute information of the content of the region to be encoded, and the attribute information includes at least one of content identifier, content name, or content display level.

[0109] In one possible implementation, the decision module is specifically used to determine the processing mode corresponding to the third information as the processing mode of the region to be encoded based on a pre-set correspondence.

[0110] In one possible implementation, the region to be encoded includes the entire area of ​​the current desktop or the area corresponding to at least one application or window on the current desktop.

[0111] In one possible implementation, the encoding module is specifically used to obtain a rendering instruction when the processing mode of the region to be encoded is the first processing mode, the rendering instruction corresponding to the content of the region to be encoded; and to encode the rendering instruction to obtain the bitstream.

[0112] In one possible implementation, the encoding module is further configured to obtain resources corresponding to the content of the region to be encoded in the first processing mode according to the rendering instructions, the resources including at least one image and / or at least one video; and to encode the resources corresponding to the content of the region to be encoded in the first processing mode to obtain the bitstream.

[0113] In one possible implementation, the encoding module is specifically configured to, when the processing mode of the region to be encoded is the second processing mode, acquire resources corresponding to the content of the region to be encoded in the second processing mode, the resources including a desktop image sequence; and encode the resources corresponding to the content of the region to be encoded in the second processing mode to obtain the bitstream.

[0114] In one possible implementation, the encoding module is further configured to acquire rendering instructions, which correspond to the content of the region to be encoded; and to encode the rendering instructions to obtain the bitstream.

[0115] In one possible implementation, the bitstream further includes processing mode information, which is used to indicate the processing mode of the region to be encoded.

[0116] In one possible implementation, the first processing mode is a processing mode based on rendering data.

[0117] In one possible implementation, the second processing mode is a desktop image sequence-based processing mode.

[0118] In one possible implementation, the bitstream further includes first data type information, which is used to indicate the bitstream corresponding to the rendering instruction.

[0119] In one possible implementation, the bitstream further includes second data type information and / or third data type information, wherein the second data type information is used to indicate the bitstream corresponding to the at least one image, and the third data type information is used to indicate the bitstream corresponding to the at least one video.

[0120] In one possible implementation, the bitstream further includes fourth data type information, which is used to indicate the bitstream corresponding to the desktop image sequence.

[0121] In one possible implementation, the bitstream corresponding to the rendering instruction is located at the end of the bitstream of the region to be encoded.

[0122] In one possible implementation, the bitstream also includes the content information.

[0123] In one possible implementation, the rendering instructions and the at least one image and / or at least one video come from the system's rendering service.

[0124] Fourthly, this application provides a hybrid decoding device, comprising: a receiving module for receiving a bitstream; a determining module for determining a processing mode for a region to be decoded based on the bitstream, the processing mode including a first processing mode or a second processing mode; an acquiring module for acquiring data corresponding to the content of the region to be decoded based on the bitstream; and a rendering module for acquiring a screen to be displayed based on the processing mode of the region to be decoded and the data corresponding to the content of the region to be decoded.

[0125] In one possible implementation, the bitstream includes processing mode information, which indicates the processing mode of the region to be decoded; correspondingly, the processing mode of the region to be decoded is determined based on the processing mode information.

[0126] In one possible implementation, the determining module is specifically used to obtain content information and data type information of the region to be decoded based on the bitstream. The content information of the region to be decoded is used to indicate the content of the region to be decoded, and the data type information is used to indicate the bitstream of rendering instructions, the bitstream of images, the bitstream of videos, or the bitstream of desktop image sequences. The module then determines the processing mode of the region to be decoded based on the content information and the data type information.

[0127] In one possible implementation, the data corresponding to the content of the area to be decoded includes rendering instructions; the rendering module is specifically used to render the image to be displayed according to the rendering instructions when the processing mode of the area to be decoded is the first processing mode.

[0128] In one possible implementation, the data corresponding to the content of the area to be decoded includes rendering instructions, and at least one image and / or at least one video; the rendering module is specifically used to render the at least one image and / or the at least one video according to the rendering instructions to obtain the screen to be displayed when the processing mode of the area to be decoded is the first processing mode.

[0129] In one possible implementation, the data corresponding to the content of the area to be decoded includes a desktop image sequence; the rendering module is specifically used to use the desktop image sequence as the screen to be displayed when the processing mode of the area to be decoded is the second processing mode.

[0130] In one possible implementation, the data corresponding to the content of the area to be decoded includes a desktop image sequence and rendering instructions; the rendering module is specifically used to render the desktop image sequence according to the rendering instructions to obtain the screen to be displayed.

[0131] In one possible implementation, the first processing mode is a processing mode based on rendering data.

[0132] In one possible implementation, the second processing mode is a desktop image sequence-based processing mode.

[0133] Fifthly, this application provides a bitstream, the bitstream comprising a first bitstream obtained based on a first processing mode and a second bitstream obtained based on a second processing mode.

[0134] In one possible implementation, the first processing mode is a processing mode based on rendering data, and the second processing mode is a processing mode based on desktop image sequences.

[0135] Sixthly, this application provides a terminal device, comprising: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described in any one of the first to second aspects above.

[0136] In a seventh aspect, this application provides a computer-readable storage medium including a computer program that, when executed on a computer, causes the computer to perform the method described in any one of the first to second aspects above.

[0137] Eighthly, this application provides a computer program product comprising computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the first to second aspects. Attached Figure Description

[0138] Figure 1 is an exemplary schematic diagram of the screen projection application scenario 100 of this application;

[0139] Figure 2 is an exemplary structural diagram of the terminal device 200 of this application;

[0140] Figure 3 is a flowchart of the hybrid coding method 300 provided in this application;

[0141] Figure 4 is a schematic diagram of the region to be encoded in this application;

[0142] Figure 5 is a flowchart of the hybrid decoding method 500 provided in this application;

[0143] Figure 6 is an exemplary schematic diagram of the hybrid codec architecture of this application;

[0144] Figure 7a is an exemplary schematic diagram of the screen projection terminal of this application;

[0145] Figure 7b is an exemplary schematic diagram of the receiving end of this application;

[0146] Figure 8 is a structural schematic diagram of the hybrid coding device 800 of this application;

[0147] Figure 9 is a schematic diagram of the structure of the hybrid decoding device 900 of this application. Detailed Implementation

[0148] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0149] The terms "first," "second," etc., used in the specification, embodiments, claims, and drawings of this application are for distinguishing purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0150] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0151] First, let's explain the key terms in this application:

[0152] Rendering: Also known as graphics rendering, it refers to the process of converting three-dimensional (3D) graphic model data into graphics that can be visualized on a two-dimensional (2D) display screen.

[0153] Rendering data: Also known as graphics rendering data, this refers to all the data required by the graphics rendering module during rendering operations, including rendering commands and rendering resources. During the rendering process, some scenes may only require rendering commands and not rendering resources.

[0154] Rendering instructions: also known as graphics rendering instructions, refer to instructions that can be understood by the graphics rendering module and can be executed for rendering operations.

[0155] Rendering resources: refers to the resource data that the graphics rendering module may use during the execution of rendering instructions, including image resources, graphics buffer queues, etc.

[0156] Buffer queue: Also known as the graphics buffer queue, this queue contains multiple buffers used to store image sequences (videos) to be rendered, which may be used during the rendering process. Images in the buffer queue can be rendered according to rendering instructions and then sent to the screen for display.

[0157] A frame refers to all the data available at a given moment to render the content to be displayed at that moment. For graphics rendering data encoding methods, this data can include all rendering instructions and resources that can render the current frame; for video encoding methods, this data can include one image in the video sequence that corresponds to the current moment and is to be displayed.

[0158] Video encoding: The process of compressing an image sequence into a bitstream.

[0159] Video decoding: The process of restoring a video stream into a reconstructed image according to specific syntax rules and processing methods.

[0160] Video coding: a general term for video encoding and video decoding, with the same Chinese translation as video encoding.

[0161] Image reconstruction: The source image is encoded to obtain a distorted or undistorted image, which can be reconstructed by the corresponding decoder.

[0162] Network bandwidth: The amount of data that a channel can transmit per unit of time, representing the network's transmission capacity.

[0163] Screen casting terminal: In screen casting application scenarios, the screen casting terminal refers to the terminal device that generates the screen casting content and initiates the screen casting.

[0164] Receiver: In screen mirroring applications, the receiver refers to the terminal device that receives and displays the content being mirrored.

[0165] Screen casting: Screen casting modes include mirror casting and non-mirror casting. In mirror casting mode, the content displayed on the receiving end is the same as the desktop content displayed on the casting end, which is analogous to mirroring the screen of the casting end to the receiving end. In non-mirror casting mode, the content displayed on the receiving end comes from the casting end, but the desktop content displayed on the casting end may be different from the content displayed on the receiving end.

[0166] Figure 1 is an exemplary schematic diagram of the screen casting application scenario 100 of this application. As shown in Figure 1, the screen casting application scenario 100 is that the screen casting end sends the screen casting content (e.g., the desktop content of the screen casting end) to the receiving end, and the receiving end displays the screen casting content.

[0167] The projection device can be a mobile phone, tablet, wearable device with wireless communication capabilities (such as a smartwatch), location tracker with positioning capabilities, computer with wireless transceiver capabilities, virtual reality (VR) device, augmented reality (AR) device, wireless device in industrial control, wireless device in self-driving, wireless device in remote medical care, wireless device in smart grid, wireless device in transportation safety, wireless device in smart city, wireless device in smart home, etc., and this application does not limit it.

[0168] The receiving end can be a smart TV, TV box, projection screen, etc., and this application does not limit it.

[0169] The network between the projection end and the receiving end can be a communication network that supports short-range communication technologies, such as a communication network that supports Wireless-Fidelity (WIFI) technology; or a communication network that supports Bluetooth technology; or a communication network that supports Near Field Communication (NFC) technology; etc.; or the communication network can also be a communication network that supports fourth-generation (4G) access technology, such as Long Term Evolution (LTE) access technology; or the communication network can also be a communication network that supports fifth-generation (5G) access technology, such as New Radio (NR) access technology; or the communication network can also be a communication network that supports third-generation (3G) access technology, such as Universal Mobile Telecommunications System (UMTS) access technology; or the communication network can also be a communication network that supports multiple wireless technologies, such as a communication network that supports LTE and NR technologies; or the communication network can also be applicable to future-oriented communication technologies, which are not specifically limited in this application.

[0170] The screen mirroring application scenarios of this application may include screen mirroring from mobile phone desktop to TV, conference presentation, remote desktop, PC / tablet desktop collaborative office, etc., and this application does not make specific limitations in this regard.

[0171] Figure 2 is an exemplary structural diagram of the terminal device 200 of this application. The terminal device 200 can be used as the aforementioned projection terminal or the aforementioned receiving terminal. As shown in Figure 2, the terminal device 200 includes: an application processor 201, a microcontroller unit (MCU) 202, a memory 203, a modem 204, a radio frequency (RF) module 205, a wireless-fidelity (Wi-Fi) module 206, a Bluetooth module 207, a sensor 208, an input / output (I / O) device 209, a positioning module 210, and other components. These components can communicate through one or more communication buses or signal lines. The aforementioned communication bus or signal line can be the CAN bus provided in this application. Those skilled in the art will understand that the terminal device 200 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0172] The following is a detailed description of each component of the terminal device 200, with reference to Figure 2:

[0173] The application processor 201 is the control center of the terminal device 200, connecting various components of the terminal device 200 via various interfaces and buses. In some embodiments, the processor 201 may include one or more processing units.

[0174] Memory 203 stores computer programs, such as operating system 211 and application program 212 as shown in Figure 2. Application processor 201 is configured to execute the computer programs in memory 203 to implement the functions defined by the computer programs. For example, application processor 201 executes operating system 211 to implement various functions of the operating system on terminal device 200. Memory 203 also stores other data besides computer programs, such as data generated during the operation of operating system 211 and application program 212. Memory 203 is a non-volatile storage medium, generally including main memory and secondary storage. Main memory includes, but is not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), or cache. Secondary storage includes, but is not limited to, flash memory, hard disk, optical disk, Universal Serial Bus (USB) disk, etc. Computer programs are usually stored on secondary storage, and the processor loads the program from secondary storage into main memory before executing the computer program.

[0175] The memory 203 can be independent and connected to the application processor 201 via a bus; the memory 203 can also be integrated with the application processor 201 into a chip subsystem.

[0176] MCU 202 is a coprocessor used to acquire and process data from sensor 208. MCU 202 has lower processing power and power consumption than application processor 201, but it features "always on," allowing it to continuously collect and process sensor data while application processor 201 is in sleep mode, ensuring normal sensor operation with extremely low power consumption. In one embodiment, MCU 202 can be a sensor hub chip. Sensor 208 may include a light sensor and a motion sensor. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of display 2091 according to the ambient light level, and the proximity sensor can turn off the display power when the terminal device 200 is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. Sensor 208 may also include other sensors such as a gyroscope, barometer, hygrometer, thermometer, and infrared sensor, which will not be described in detail here. The MCU 202 and sensor 208 can be integrated onto the same chip or are separate components connected via a bus.

[0177] The modem 204 and the radio frequency (RF) module 205 constitute the communication subsystem of the terminal device 200, used to implement the main functions of the wireless communication standard protocol. The modem 204 is used for encoding / decoding, signal modulation / demodulation, and equalization. The RF module 205 is used for receiving and transmitting wireless signals, and includes, but is not limited to, an antenna, at least one amplifier, a coupler, and a duplexer. The RF module 205 works with the modem 204 to implement wireless communication functions. The modem 204 can be a standalone chip or integrated with other chips or circuits to form a system-on-a-chip (SoC) or integrated circuit. These chips or integrated circuits can be used in all devices that implement wireless communication functions, including: mobile phones, computers, laptops, tablets, routers, wearable devices, automobiles, and home appliances.

[0178] The terminal device 200 can also use Wi-Fi module 206, Bluetooth module 207, etc., for wireless communication. Wi-Fi module 206 provides network access to the terminal device 200 in accordance with Wi-Fi related standard protocols. The terminal device 200 can access the Wi-Fi access point through Wi-Fi module 206 and thus access the Internet. In some other embodiments, Wi-Fi module 206 can also act as a Wi-Fi wireless access point, providing Wi-Fi network access to other devices. Bluetooth module 207 enables short-range communication between the terminal device 200 and other devices (such as mobile phones, smartwatches, etc.). In this embodiment, Wi-Fi module 206 can be an integrated circuit or a Wi-Fi chip, and Bluetooth module 207 can be an integrated circuit or a Bluetooth chip.

[0179] The positioning module 210 is used to determine the geographical location of the terminal device 200. It is understood that the positioning module 210 can specifically be a receiver for a global positioning system (GPS), such as the BeiDou Navigation Satellite System or the Russian GLONASS system.

[0180] The Wi-Fi module 206, Bluetooth module 207, and positioning module 210 can each be a separate chip or integrated circuit, or they can be integrated together. For example, in one embodiment, the Wi-Fi module 206, Bluetooth module 207, and positioning module 210 can be integrated onto the same chip. In another embodiment, the Wi-Fi module 206, Bluetooth module 207, positioning module 210, and MCU 202 can also be integrated into the same chip.

[0181] Input / output devices 209 include, but are not limited to: display 2091, touch screen 2092, and audio circuit 2093, etc.

[0182] The touchscreen 2092 can collect touch events from the user of the terminal device 200 on or near the touchscreen 2092 (such as user actions using a finger, stylus, or any suitable object on or near the touchscreen 2092) and send the collected touch events to other devices (such as the application processor 201). User actions near the touchscreen 2092 can be termed hover touch; through hover touch, the user can select, move, or drag objects (such as icons) without directly touching the touchscreen 2092. Furthermore, the touchscreen 2092 can be implemented using various types of touchscreens, including resistive, capacitive, infrared, and surface acoustic wave.

[0183] The display (also called a screen) 2091 is used to display information input by the user or information shown to the user. The display can be configured using a liquid crystal display (LCD), an organic light-emitting diode (OLED), or other similar methods. The touchscreen 2092 can cover the display 2091. When the touchscreen 2092 detects a touch event, it transmits the information to the application processor 201 to determine the type of touch event. The application processor 201 then provides corresponding visual output on the display 2091 based on the type of touch event. Although in Figure 2, the touchscreen 2092 and the display 2091 are shown as two separate components implementing the input and output functions of the terminal device 200, in some embodiments, the touchscreen 2092 and the display 2091 can be integrated to achieve the input and output functions of the terminal device 200. Furthermore, the touchscreen 2092 and the display 2091 can be configured as a full-panel display on the front of the terminal device 200 to achieve a borderless structure.

[0184] Audio circuit 2093, speaker 2094, and microphone 2095 provide an audio interface between the user and terminal device 200. Audio circuit 2093 converts received audio data into electrical signals and transmits them to speaker 2094, where speaker 2094 converts them into sound signals for output. On the other hand, microphone 2095 converts collected sound signals into electrical signals, which are received by audio circuit 2093, converted into audio data, and then transmitted to another device via modem 204 and radio frequency module 205, or output to memory 203 for further processing.

[0185] In addition, the terminal device 200 may also have fingerprint recognition functionality. For example, a fingerprint acquisition device can be configured on the back of the terminal device 200 (e.g., below the rear camera) or on the front of the terminal device 200 (e.g., below the touchscreen 2092). Alternatively, the fingerprint acquisition device can be integrated into the touchscreen 2092 to implement fingerprint recognition functionality; that is, the fingerprint acquisition device can be integrated with the touchscreen 2092 to achieve the fingerprint recognition function of the terminal device 200. In this case, the fingerprint acquisition device is configured in the touchscreen 2092, and may be part of the touchscreen 2092 or configured in the touchscreen 2092 in other ways. The main component of the fingerprint acquisition device in this embodiment is a fingerprint sensor, which can employ any type of sensing technology, including but not limited to optical, capacitive, piezoelectric, or ultrasonic sensing technologies.

[0186] Furthermore, the operating system 211 mounted on the terminal device 200 can provide... This application does not impose any restrictions on other operating systems, including those used in the embodiments of the present application.

[0187] With Taking the terminal device 200 with an operating system as an example, the terminal device 200 can be logically divided into a hardware layer, an operating system 211, and an application layer. The hardware layer includes hardware resources such as the application processor 201, MCU 202, memory 203, modem 204, Wi-Fi module 206, sensor 208, and positioning module 210, as described above. The application layer includes one or more applications, such as application 212, which can be any type of application, such as a social application, e-commerce application, or browser. The operating system 211, as software middleware between the hardware layer and the application layer, is a computer program that manages and controls hardware and software resources.

[0188] In one embodiment, the operating system 211 includes a kernel, a hardware abstraction layer (HAL), libraries and runtime, and a framework. The kernel provides low-level system components and services, such as power management, memory management, thread management, and hardware drivers. Hardware drivers include Wi-Fi drivers, sensor drivers, and positioning module drivers. The HAL encapsulates the kernel drivers, providing interfaces to the framework and shielding them from low-level implementation details. The HAL runs in user space, while the kernel drivers run in kernel space.

[0189] Libraries and runtimes, also known as runtime libraries, provide the necessary library files and execution environment for executable programs at runtime. In one embodiment, libraries and runtimes include the Android runtime (ART), libraries, and scene package runtimes. ART is a virtual machine or virtual machine instance capable of converting application bytecode into machine code. Libraries are program libraries that provide support for executable programs at runtime, including browser engines (such as WebKit), script execution engines (such as JavaScript engines), and graphics processing engines. The scene package runtime is the runtime environment for scene packages, mainly including the page execution environment and the script execution environment. The page execution environment parses page code in HTML, CSS, and other formats by calling corresponding libraries, while the script execution environment parses and executes code or executable files implemented in scripting languages ​​such as JavaScript by calling corresponding function libraries.

[0190] The framework provides various basic public components and services for applications in the application layer, such as window management, location management, and so on. In one embodiment, the framework includes a geofencing service, a policy service, a notification manager, and so on.

[0191] The functions of each component of the operating system 211 described above can be implemented by the application processor 201 executing the program stored in the memory 203.

[0192] Those skilled in the art will understand that the terminal device 200 may include fewer or more components than those shown in FIG2, which only includes components more relevant to the various implementations disclosed in this application.

[0193] Based on the aforementioned screen projection application scenario 100 and terminal device 200, this application provides a hybrid encoding and decoding method and apparatus to solve the problem that the large data volume leads to a high transmission bit rate, which in turn affects the screen projection effect.

[0194] This application involves two processing modes:

[0195] 1. Processing mode based on rendering data

[0196] The projection device encodes the rendering instructions of the application (APP) or system service. If necessary, it can also encode the rendering resources required to execute the aforementioned rendering instructions. The encoded bitstream is transmitted to the receiving device. The receiving device decodes the bitstream to obtain the rendering instructions and the rendering resources. Optionally, if only the rendering instructions are available, the receiving device can perform rendering locally based on the rendering instructions (bitstream without rendering resources), or obtain the rendering resources required to execute the aforementioned rendering instructions locally. Then, the receiving device renders the screen based on the aforementioned rendering data (including rendering instructions and rendering resources) and displays it on the receiving device's monitor.

[0197] In this processing mode, the encoded objects include rendering instructions, and, if necessary, rendering resources required to execute the rendering instructions. Rendering resources may include images and / or videos. The aforementioned rendering instructions and rendering resources can be collectively referred to as rendering data.

[0198] 2. Desktop image sequence-based processing mode

[0199] The projection device uses video encoding to encode the desktop image sequence into a video stream, which is then transmitted to the receiving device. The receiving device decodes the video stream to obtain the reconstructed desktop image sequence, which is then displayed on the receiving device's monitor. The receiving device can also perform a lightweight rendering of the reconstructed desktop image sequence for better playback.

[0200] In this processing mode, the encoded object includes a desktop image sequence, which includes one or more frames of images refreshed on the desktop of the projection terminal.

[0201] Figure 3 is a flowchart of process 300 of the hybrid encoding method provided in this application. Process 300 can be executed by the projection terminal described above. Process 300 is described as a series of steps or operations. It should be understood that process 300 can be executed in various orders and / or occur simultaneously, and is not limited to the execution order shown in Figure 3. Process 300 may include:

[0202] Step 301: Obtain the content information of the region to be encoded.

[0203] The area to be encoded can include the entire area of ​​the current desktop or the area corresponding to at least one application or window on the current desktop. The current desktop refers to the screen of the projection device at the current moment. Based on this, the entire area of ​​the current desktop can refer to the entire screen of the projection device; while the area corresponding to at least one application or window on the current desktop can refer to the area occupied by at least one application or window displayed on the current desktop on the screen of the projection device. This area can occupy the entire screen of the projection device (full-screen display) or a portion of the screen of the projection device (small window or split-screen display).

[0204] For example, Figure 4 is a schematic diagram of the area to be encoded in this application. As shown in Figure 4, the upper part of the current desktop displays the video in the playback window of the video application, and the lower part displays the system main interface. If the area to be encoded refers to the entire area of ​​the current desktop, then the area to be encoded is the entire screen of the projection terminal. In this case, the content of the area to be encoded includes the content of the playback window of the video application (e.g., video, controls, icons, etc.) and the system main interface. If the area to be encoded refers to the window of the video application on the current desktop, then the area to be encoded is a portion of the screen of the projection terminal. In this case, the content of the area to be encoded only includes the content of the window of the video application. It should be understood that when the area to be encoded refers to the area corresponding to at least one application or window on the current desktop, the size of the area to be encoded depends on the size of the area occupied by at least one application or window. When at least one application or window occupies the entire screen, the area to be encoded is the entire area of ​​the current desktop. When at least one application or window occupies a portion of the screen, the area to be encoded is a portion of the current desktop. This application does not specifically limit the size of the area to be encoded in this case.

[0205] The content information of the region to be encoded is used to indicate the content of the region to be encoded. In this application, the content information may include the following two types:

[0206] (1) The content information includes first information and second information. The first information includes resource information of the content of the region to be encoded in the first processing mode, and the second information includes resource information of the content of the region to be encoded in the second processing mode.

[0207] In one possible implementation, in the first processing mode, the content of the region to be encoded includes images and / or video within the region to be encoded. Accordingly, the first information includes image information and / or video information, wherein the image information includes the number of images (N). img ) or image resolution (IW i ×IH i {i=1,2,3,…,N img The video information includes at least one of the following: (also known as image resolution), and the video information includes the number of videos (N). buf Since videos are typically stored in a buffer in a queue, this is also referred to as the number of buffer queues, and the video resolution (BW). j ×BH j {j=1,2,3,…,N buf}), the display resolution corresponding to the video (DW) k ×DH k {k = 1, 2, 3, ..., N} bif}) or the corresponding encoding frame rate of the video (FR) m {m=1,2,3,…,N buf At least one of the following:}). For example, as shown in Figure 4, the area to be encoded is the entire area of ​​the current desktop, and the content of the area to be encoded may include videos in the playback window of video applications located at the top of the current desktop, and icons in the system main interface located at the bottom of the current desktop. Accordingly, the first information may include the number of icons, the resolution of the icons, and the number of videos, the resolution of the videos, the encoding frame rate of the videos, etc.

[0208] In one possible implementation, in the second processing mode, the content of the region to be encoded includes a sequence of desktop images of the region to be encoded, and correspondingly, the second information includes the resolution (W) of the desktop images in the desktop image sequence. s ×H s ), the refresh rate of the desktop image sequence (F) ps ) or the encoded frame rate (F) of a desktop image sequence r At least one of the following. For example, as shown in Figure 4, the area to be encoded is the entire area of ​​the current desktop. The screen of the projection terminal refreshes at a certain refresh rate, and the content of the area to be encoded may include a sequence of desktop images composed of desktop images refreshed within a certain time period. Correspondingly, the second information may include the resolution of the aforementioned desktop images, the refresh rate of the desktop image sequence, the encoding frame rate of the desktop image sequence, etc.

[0209] Optionally, the first processing mode can be the processing mode based on the rendered data described above.

[0210] Optionally, the second processing mode can be the desktop image sequence-based processing mode described above.

[0211] It should be noted that the above exemplary description of the data contained in the first information and the second information does not constitute a limitation. The data contained in the first information and the second information mainly depend on the resources required by the corresponding processing mode. Therefore, this application does not specifically limit the first information and the second information.

[0212] (2) The content information includes third information, which includes the attribute information of the content of the area to be encoded, including at least one of the content identifier, content name, or content display level.

[0213] The third piece of information includes, for example, the name, identifier, and display hierarchy of the application / system window program. For instance, as shown in Figure 4, the third piece of information includes the name and display hierarchy of the video application displayed at the top of the current desktop.

[0214] It should be noted that the above examples describe the data contained in the third information, but this does not constitute a limitation. The data contained in the third information mainly depends on the rendering object (application / system window program), so this application does not make specific limitations on the third information.

[0215] In this application, the content information can be obtained from the rendering information of the application or system window program being rendered, which is received in the rendering service (e.g., the system's rendering service or the application's rendering service), or it can be obtained by reading the current system task manager, without any specific limitation.

[0216] Step 302: Decision on the processing mode of the region to be encoded based on the content information.

[0217] The projection device can decide the processing mode of the area to be encoded in the following two ways:

[0218] (1) Compare the benefits of the first processing mode and the second processing mode based on the first information and the second information; determine the processing mode of the region to be encoded as the one with the greater benefit between the first processing mode and the second processing mode.

[0219] Benefits can refer to the advantageous effects obtained after processing by the first or second processing mode. Therefore, significant benefits can include, for example, low encoding bitrate, low power consumption, and low complexity. In other words, the mode with the lower encoding bitrate, lower power consumption, or lower complexity among the first and second processing modes can be determined as the processing mode for the region to be encoded.

[0220] In one possible implementation, the benefit of the second processing mode is determined to be greater than the benefit of the first processing mode when at least one of the following first conditions is met; the benefit of the first processing mode is determined to be greater than the benefit of the second processing mode when none of the first conditions are met. The first conditions include the following:

[0221] ① When the number of videos is greater than 0 (N) buf When W > 0), the resolution of the desktop image is less than or equal to the sum of the resolutions of multiple videos (which can be videos in the buffer queue); s ×H s ≤∑ j (BW j ×BH j )

[0222] j≥1.

[0223] When this condition is met, the second processing mode is expected to save more bitrate than the first processing mode, so the second processing mode is more profitable.

[0224] ② When the number of videos is greater than 0 (N) buf When the value is greater than 0, the sum of the resolutions of multiple videos (which may be videos in the cache queue) is less than the resolution of the desktop image, and the sum of the resolutions of the multiple videos is greater than the product of the sum of the display resolutions corresponding to the multiple videos and the first preset value; W s ×H s >∑ j (BW j ×BH j )>Σ k (DW k ×DH k )×R0

[0225] j≥1; k≥1; R0 is a predefined constant, optional, 1≤R0≤2.

[0226] When this condition is met, the second processing mode is expected to save more bitrate than the first processing mode, so the second processing mode is more profitable.

[0227] ③ The number of videos is greater than the second preset value (N) buf >N0), the second preset value is a positive integer (N0≥1);

[0228] When this condition is met, the first processing mode requires encoding multiple videos, thus necessitating the execution of multiple video encoders. This is anticipated to exceed the real-time encoding capabilities of the video encoders, increasing the system's processing burden. In contrast, the second processing mode only requires a single video encoder, resulting in lower system complexity and lower power consumption; therefore, the second processing mode offers greater benefits.

[0229] ④ When the number of videos is greater than 0 (N) buf When the resolution of multiple images is greater than 0, the sum of their resolutions is greater than the sum of the resolutions of the desktop images refreshed within a preset time period (e.g., 1 / F0 seconds). s ×H s ×F ps / F0≤Σ i (IW i ×IH i )

[0230] i≥1; F0 is a positive integer constant, optional, 0 <F0≤10。

[0231] When this condition is met, the first processing mode may generate a large instantaneous bitrate. Using the second processing mode is expected to save bitrate and provide a smoother, more lag-free display. Therefore, the second processing mode is more beneficial.

[0232] ⑤ When the number of videos is greater than 0 (N) buf When the sum of the encoded frame rates of the video is greater than the refresh rate of the desktop image, or the sum of the encoded frame rates of the video is greater than the encoded frame rate of the desktop image sequence. ps <∑ m FR m

[0233] Or, F r <∑ m FR m

[0234] m≥1.

[0235] When this condition is met, the first processing mode may generate greater power consumption and bit rate. Using the second processing mode is expected to result in lower power consumption and lower bit rate, thus the second processing mode is more beneficial.

[0236] Based on this, when one of the above five conditions is met, the processing mode for the region to be encoded can be determined as the second processing mode. Conversely, when none of the above five conditions are met, the processing mode for the region to be encoded can be determined as the first processing mode.

[0237] Optionally, when the number of videos (N) buf When =0), the processing mode of the region to be encoded can be decided as the first processing mode.

[0238] Optionally, when there are no rendering resources (i.e., corresponding to the number N images in the content information), img =0 and the number of videos N buf When =0), the processing mode of the region to be encoded can be decided as the first processing mode.

[0239] (2) Based on the pre-set correspondence, the processing mode corresponding to the third information is determined as the processing mode of the region to be encoded.

[0240] The third piece of information can indicate the application / system window program currently on the desktop. Based on this, the application scenario of the current desktop can be inferred, and the processing mode of the area to be coded can be directly determined. For example, if the current desktop displays a browser-type application, it can be inferred that it contains a lot of text and images, so the first processing mode can be used; or, if the current desktop displays an office software-type application, it can be inferred that it mainly contains text and images and no video, so the first processing mode can be used; or, if the current desktop only contains the system interface and does not display any applications, the first processing mode can be used; or, if the current desktop displays a video playback-type application, it can be inferred that it contains video playback, so the second processing mode can be used.

[0241] It should be noted that the decision-making methods of the two processing modes mentioned above are only examples. In specific implementation, this application may also consider other decision-making methods to determine the processing mode of the region to be encoded as the one with greater benefit between the first processing mode and the second processing mode. Therefore, no specific limitation is made on the decision-making method.

[0242] Step 303: Encode the content of the region to be encoded according to the processing mode of the region to be encoded to obtain the bitstream.

[0243] Based on the decision results regarding the processing mode of the region to be encoded, the projection terminal can include the following situations when encoding the content of the region to be encoded:

[0244] 1. The processing mode for the region to be encoded is the first processing mode.

[0245] 1.1 The projection terminal obtains the rendering instruction, which corresponds to the content of the area to be encoded; the rendering instruction is encoded to obtain the bitstream.

[0246] Rendering commands can originate from applications, system windows, etc., on the current desktop of the projection device. The projection device can capture user operation commands from the screen or other input modules, then generate rendering commands in response to these operation commands and send them to the rendering service. In this way, rendering commands can be obtained from the rendering service.

[0247] In this application, the rendering service intercepts rendering instructions generated by all applications and system window programs running in the system, and can then render the image of the current desktop based on these rendering instructions.

[0248] 1.2 In addition to encoding rendering instructions, the projection terminal can also encode rendering resources, that is, obtain the resources corresponding to the content of the area to be encoded in the first processing mode according to the rendering instructions. The resources include at least one image and / or at least one video. The resources corresponding to the content of the area to be encoded in the first processing mode are encoded to obtain the bitstream.

[0249] The resources corresponding to the content of the region to be encoded (also known as rendering resources) can be obtained based on the parsing of rendering instructions. Rendering resources contain at least one image and / or at least one video (video can be stored in a buffer as a queue). Rendering resources occupy a large amount of storage space, and applications or system window programs typically include these resources in the rendering instructions with the most economical signal data (e.g., index numbers) and send them to the rendering service. During the rendering process, the rendering service parses the rendering instructions to obtain the aforementioned signal data, and then locates these rendering resources in the system path, application, or system window program based on this signal data before performing the rendering.

[0250] In this application, in addition to the above-mentioned process of obtaining rendering resources, if the application or system window program directly sends the required rendering resources to the rendering service, the projection terminal can also directly obtain the rendering resources from the rendering service.

[0251] In the first processing mode, rendering instructions need to be encoded. If necessary, resources (including images and / or videos) corresponding to the content of the area to be encoded also need to be encoded. Rendering instructions can be encoded using data encoding methods, images can be encoded using image encoding methods, and videos can be encoded using a video encoder. This application does not limit the specific implementation of the aforementioned encoding methods.

[0252] 2. The processing mode for the region to be encoded is the second processing mode.

[0253] 2.1 The projection terminal acquires the resources corresponding to the content of the area to be encoded in the second processing mode, which includes a desktop image sequence; the resources corresponding to the content of the area to be encoded in the second processing mode are encoded to obtain a bitstream.

[0254] In this application, the resources corresponding to the content of the region to be encoded can be obtained by rendering the rendering data once, or by accessing the memory, without any specific limitation.

[0255] Accordingly, the resolution of the desktop images in the desktop image sequence included in the second information above can be obtained by parsing the rendering instructions or by obtaining them from the system, without any specific limitation; the refresh rate or the encoding frame rate of the desktop image sequence can be obtained from the rendering service or by obtaining them from the system, without any specific limitation.

[0256] 2.2 In addition to encoding the resources corresponding to the content of the area to be encoded, the projection terminal also encodes rendering instructions, that is, it obtains rendering instructions, which correspond to the content of the area to be encoded; and encodes the rendering instructions to obtain the bitstream.

[0257] In the second processing mode, resources corresponding to the content of the area to be encoded (including desktop image sequences) need to be encoded. If necessary, rendering instructions also need to be encoded (to ensure the continuity of rendering instructions; if not continuous, more rendering data needs to be encoded and sent when switching from the second processing mode to the first processing mode in order for the image to be correctly rendered at the receiving end). The desktop image sequence can be encoded using a video encoder, and the rendering instructions can be encoded using data encoding. This application does not limit the specific implementation of the aforementioned encoding methods. It should be noted that in the second processing mode, the projection end may not encode the rendering instructions, but only encode the resources corresponding to the content of the area to be encoded (including desktop image sequences). In this case, when switching from the second processing mode to the first processing mode, the projection end may need to encode more rendering instruction information so that the receiving end can correctly parse the rendering instructions.

[0258] Optionally, the bitstream may also include processing mode information, which indicates the processing mode of the region to be encoded.

[0259] In this way, the receiving end can obtain the processing mode of the region to be decoded by parsing the bitstream. This processing mode can be either the first processing mode or the second processing mode.

[0260] In one possible implementation, the projection end and the receiving end can inform the receiving end of the processing mode of the area to be decoded through an agreed-upon method. For example, the projection end places the bitstream of the rendering instructions in a frame at the end of the frame's bitstream (for example, a frame's bitstream of the area to be encoded may include the bitstream of the rendering instructions and the bitstream of the resources corresponding to the content of the area to be encoded; in this case, the bitstream of the resources corresponding to the content of the area to be encoded is placed first, followed by the bitstream of the rendering instructions; if there are no other bitstreams, then the bitstream of the rendering instructions is at the end of the bitstream of the area to be encoded in that frame). When the receiving end parses the bitstream of the rendering instructions in a frame, it determines whether the bitstream of that frame contains the bitstream of an image / video. If not, the processing mode of the area to be decoded in that frame is the first processing mode; if it does, it further determines whether the resolution of the image / video is the same as the resolution of the desktop image of the area to be decoded (to achieve this determination, the projection end can write the above content information into the bitstream). If they are the same, the processing mode of the area to be decoded in that frame is the second processing mode; if they are different, the processing mode of the area to be decoded in that frame is the first processing mode. It should be noted that, in addition to the above examples, the projection end and the receiving end may also use other agreed-upon methods to inform the receiving end of the processing mode of the area to be decoded, and this application does not impose specific limitations on this.

[0261] Optionally, the bitstream may also include first data type information, which is used to indicate the bitstream corresponding to the rendering instructions.

[0262] Optionally, the bitstream may also include second data type information and / or third data type information, wherein the second data type information is used to indicate the bitstream corresponding to at least one image, and the third data type information is used to indicate the bitstream corresponding to at least one video.

[0263] Optionally, the bitstream may also include fourth data type information, which is used to indicate the bitstream corresponding to the desktop image sequence.

[0264] In one possible implementation, based on at least one of the aforementioned first, second, third, and fourth data types, the projection end and the receiving end can synchronize the processing mode of the area to be encoded / decoded. That is, the encoding end carries the data type information of the corresponding bitstream in the bitstream; the receiving end decompresses the bitstream, and when it obtains the fourth data type information, it can determine that the processing mode of the area to be encoded / decoded is the second processing mode; or, when it obtains the second and / or third data type information, it can determine that the processing mode of the area to be encoded / decoded is the first processing mode.

[0265] The projection device can directly write the data type corresponding to the bitstream into the bitstream, such as the bitstream of rendering instructions, the bitstream of images, the bitstream of videos, or the bitstream of desktop image sequences, so that the receiving device can parse the bitstream based on the corresponding decoding method.

[0266] In this way, the receiving end can determine the type of the bitstream by parsing the data type information in the bitstream of the area to be decoded, and then use the decoder corresponding to the projection end to decode the different types of bitstreams and reconstruct them into rendering instructions (the bitstream includes the first data type information), images (the bitstream includes the second data type information), videos (the bitstream includes the third data type information), desktop image sequences (the bitstream includes the fourth data type information), etc.

[0267] In this embodiment, the processing mode applicable to the region to be encoded is determined based on the content information of the region to be encoded. This processing mode can provide greater benefits compared to another processing mode. On the one hand, it realizes the integration of the two processing modes, and on the other hand, it can encode the content of the region to be encoded in the processing mode most suitable for it, which can reduce the encoding bitrate, reduce power consumption and complexity, and improve the projection effect.

[0268] Figure 5 is a flowchart of process 500 of the hybrid decoding method provided in this application. Process 500 can be executed by the receiving end described above. Process 500 is described as a series of steps or operations. It should be understood that process 500 can be executed in various orders and / or occur simultaneously, and is not limited to the execution order shown in Figure 5. Process 500 may include:

[0269] Step 501: Receive the bitstream.

[0270] Step 502: Determine the processing mode of the region to be decoded based on the bitstream.

[0271] The processing mode includes a first processing mode or a second processing mode. Optionally, the first processing mode can be the processing mode based on rendered data as described above. Optionally, the second processing mode can be the processing mode based on desktop image sequences as described above.

[0272] The receiving end can decide the processing mode of the region to be decoded in the following two ways:

[0273] (1) The bitstream includes processing mode information, and the processing mode of the region to be decoded is determined based on the processing mode information.

[0274] As shown in the embodiment of the projection terminal method in Figure 3, the projection terminal can directly write processing mode information into the bitstream and transmit it to the receiving terminal. This processing mode information is used to indicate the processing mode of the area to be decoded. Therefore, the receiving terminal can parse the bitstream, obtain the processing mode information from it, and then determine the processing mode of the area to be decoded based on the processing mode information.

[0275] (2) Obtain the content information and data type information of the region to be decoded based on the bitstream. The content information of the region to be decoded is used to indicate the content of the region to be decoded, and the data type information is used to indicate the bitstream of the rendering instruction, the bitstream of the image, the bitstream of the video, or the bitstream of the desktop image sequence. Determine the processing mode of the region to be decoded based on the content information and data type information of the region to be decoded.

[0276] As shown in the embodiment of the screen projection method in Figure 3, the screen projection end and the receiving end can inform the receiving end of the processing mode of the area to be decoded through an agreed-upon method. Therefore, the receiving end can determine the processing mode of the area to be decoded through the aforementioned method. For example, when the receiving end parses the bitstream of the rendering instructions in a frame, it determines whether the bitstream of that frame contains an image / video bitstream. If it does not, the processing mode of the area to be decoded in that frame is the first processing mode; if it does, it further determines whether the resolution of the image / video is the same as the resolution of the desktop image in the area to be decoded (to achieve this determination, the screen projection end can write the content information mentioned in the embodiment shown in Figure 3 into the bitstream). If they are the same, the processing mode of the area to be decoded in that frame is the second processing mode; if they are different, the processing mode of the area to be decoded in that frame is the first processing mode. It should be noted that, in addition to the above example, the screen projection end and the receiving end can also use other agreed-upon methods to inform the receiving end of the processing mode of the area to be decoded, and this application does not specifically limit this.

[0277] Optionally, the bitstream may also include first data type information, which is used to indicate the bitstream corresponding to the rendering instructions.

[0278] Optionally, the bitstream may also include second data type information and / or third data type information, wherein the second data type information is used to indicate the bitstream corresponding to at least one image, and the third data type information is used to indicate the bitstream corresponding to at least one video.

[0279] Optionally, the bitstream may also include fourth data type information, which is used to indicate the bitstream corresponding to the desktop image sequence.

[0280] The receiving end can obtain one or more of the above-mentioned data types based on the bitstream, thereby determining the data type corresponding to one or more segments of the bitstream, such as the bitstream of rendering instructions, the bitstream of images, the bitstream of videos, or the bitstream of desktop image sequences, and then parse the bitstream based on the corresponding decoding method.

[0281] Step 503: Obtain the data corresponding to the content of the region to be decoded based on the bitstream.

[0282] In the first processing mode, the projection end can encode rendering instructions, and if necessary, can also encode resources (including images and / or videos) corresponding to the content of the area to be encoded. Rendering instructions can be encoded using data encoding methods, images using image encoding methods, and videos using video encoders. Correspondingly, the receiving end can decode the rendering instruction stream using data decoding methods, decode the image stream using image decoding methods, and decode the video stream using video decoding methods. In this mode, the data corresponding to the content of the area to be decoded includes rendering instructions, and may also include images and / or videos (also known as rendering resources).

[0283] In the second processing mode, the projection end can encode resources (including desktop image sequences) corresponding to the content of the area to be encoded. If necessary, it can also encode rendering instructions. The desktop image sequence can be encoded using a video encoder, and the rendering instructions can be encoded using data encoding. This application does not limit the specific implementation of the aforementioned encoding methods. Correspondingly, the receiving end can use video decoding to decode the bitstream of the desktop image sequence to obtain the desktop image sequence, and use data decoding to decode the bitstream of the rendering instructions to obtain the rendering instructions. In this case, the data corresponding to the content of the area to be decoded includes the desktop image sequence, and may also include rendering instructions.

[0284] Step 504: Obtain the screen to be displayed based on the processing mode of the area to be decoded and the data corresponding to the content of the area to be decoded.

[0285] Based on the processing mode of the area to be decoded determined by the above steps, the receiving end can obtain the screen to be displayed in the following ways:

[0286] 1. The processing mode for the area to be decoded is the first processing mode.

[0287] 1.1. Render the image to be displayed according to the rendering instructions.

[0288] In one possible implementation, under the first processing mode, the projection end may not encode the rendering resources. In this case, the receiving end can directly render according to the rendering instructions and draw the screen to be displayed.

[0289] In one possible implementation, in the first processing mode, the projection end may not encode the rendering resources. In this case, the receiving end can obtain signal data (such as index number) by parsing the rendering instructions, and then find the corresponding rendering resources in the system path, application, or system window program based on these signal data.

[0290] The receiving end renders the rendering resources obtained through the above processing according to the rendering instructions obtained from the decompressed bitstream, and can obtain the screen to be displayed.

[0291] 1.2. Render at least one image and / or at least one video according to the rendering instructions to obtain the screen to be displayed.

[0292] The receiving end obtains rendering instructions and rendering resources (including at least one image and / or at least one video) from the decompressed bitstream, and then renders the rendering resources based on the rendering instructions to obtain the screen to be displayed.

[0293] 2. The processing mode for the area to be decoded is the second processing mode.

[0294] 2.1 Use the desktop image sequence as the screen to be displayed.

[0295] After the receiving end decompresses the bitstream and reconstructs the desktop image sequence, it can be directly used as the screen to be displayed without rendering.

[0296] 2.2 Render the desktop image sequence according to the rendering instructions to obtain the screen to be displayed.

[0297] After the receiving end decompresses the bitstream and reconstructs the desktop image sequence, it can also render the image based on rendering instructions to obtain the image to be displayed, thereby improving the visual appeal of the image to be displayed.

[0298] In this embodiment, the processing mode suitable for the area to be decoded is determined based on the bitstream transmitted by the projection end. This processing mode can provide greater benefits compared to another processing mode. On the one hand, it realizes the integration of the two processing modes, and on the other hand, it can render the content of the area to be decoded in the processing mode most suitable for it, which can improve transmission efficiency, reduce power consumption and complexity, and improve the projection effect.

[0299] The technical solution of this application will be described in detail below using several specific embodiments.

[0300] Example 1

[0301] Figure 6 is an exemplary schematic diagram of the hybrid encoding and decoding architecture of this application. As shown in Figure 6, both the projection end and the receiving end include a rendering service module (which can be a system rendering service or an application-level rendering service). In addition, the projection end also includes a projection screen, an encoding decision module, an image encoder, a video encoder, a data encoder, and an encapsulation module. The receiving end also includes a decapsulation module, an image decoder, a video decoder, a data decoder, and a receiving screen. This architecture can be used in end-to-end transmission systems, especially transmission systems involving screen encoding and graphics rendering, including but not limited to short-distance low-latency projection, multi-screen content interaction, shared desktops, and collaborative conferencing.

[0302] In this architecture, the projection screen (including other input modules) captures operation commands. These commands can be sent to the rendering service module via applications (APP), system windows, etc., to generate rendering commands that can be recognized by the rendering service module.

[0303] The rendering service module obtains rendering instructions. On one hand, it performs rendering processing and displays the results based on the rendering instructions. On the other hand, it obtains the rendering resource (including image / video) information required for rendering as indicated in the rendering instructions (corresponding to the first information above). Furthermore, it obtains desktop image sequence information (corresponding to the second information above). Then, based on the rendering resource information and desktop image sequence information, it decides whether to use the processing mode based on rendering data (corresponding to the first processing mode above) or the processing mode based on desktop image sequence (corresponding to the second processing mode above).

[0304] When using the rendering data-based processing mode, the data encoder encodes the rendering instructions to obtain the rendering instruction bitstream, the image encoder encodes the images in the rendering resources to obtain the image bitstream (optional step), and the video encoder encodes the video in the rendering resources to obtain the video bitstream (optional step).

[0305] When using the desktop image sequence-based processing mode, the data encoder encodes the rendering instructions to obtain the rendering instruction bitstream (optional step), and the video encoder encodes the desktop image sequence to obtain the video bitstream.

[0306] The encapsulation module encapsulates the bitstream obtained from the above encoding according to certain rules and transmits it to the receiving end.

[0307] The receiving end acquires the encapsulated data, and the decapsulation module decapsulates it according to the rules corresponding to the projection end to obtain the bitstream. Based on the processing mode information in the bitstream or the data in the pre-set bitstream, it determines whether to use the processing mode based on rendering data or the processing mode based on desktop image sequence.

[0308] When using a rendering data-based processing mode, the data decoder decodes the rendering instruction stream to obtain the rendering instruction, the image decoder decodes the image stream to obtain the image in the rendering resource (optional step), and the video decoder decodes the video stream to obtain the video in the rendering resource (optional step). The rendering service module, based on the rendering instruction and combined with the rendering resource (optional step), renders the screen to be displayed (including images and / or video), and the receiving end displays the screen to be displayed.

[0309] When using the desktop image sequence-based processing mode, the data decoder decodes the rendering instruction bitstream to obtain the rendering instructions (optional step), and the video decoder decodes the video bitstream to obtain the desktop image sequence. The receiving end screen displays the desktop image sequence, or the rendering service module renders the desktop image sequence to obtain the desktop image sequence to be displayed, and the receiving end screen displays the image to be displayed (including the desktop image sequence).

[0310] In this application, the step of deciding whether to use a processing mode based on rendering data or a processing mode based on desktop image sequences on the projection end can be implemented by the rendering service module or by an independent encoding decision module, without any specific limitation.

[0311] Example 2

[0312] Figure 7a is an exemplary schematic diagram of the screen projection terminal of this application. As shown in Figure 7a, the screen projection terminal can perform the following steps when encoding the entire area of ​​the current desktop (the area to be encoded):

[0313] Step 1: Obtain rendering instructions.

[0314] The source of rendering instructions can be apps, system window programs, etc., installed on the terminal device. These apps and system window programs can obtain user operation instructions from the projection terminal screen or other input modules, and then generate rendering instructions in response to these operation instructions. Based on the rendering instructions, the content of the area to be encoded can be rendered, including all desktop content displayed on the projection terminal screen, such as icons, the interface of apps / system window programs, and the content of small windows.

[0315] Step 2: Obtain the rendering resources required for the rendering process, as well as the corresponding rendering resource information.

[0316] Rendering resources can be acquired based on the parsing of rendering commands. Rendering resources can include images (e.g., icons), videos in a buffer queue (e.g., videos played by a video app), etc. Rendering resources occupy a significant amount of storage space. Generally, apps or system window programs will include these rendering resources in the rendering commands with the most economical signal data (e.g., index numbers) during transmission. During the rendering process, after parsing the rendering commands, the required rendering resources are located in the system or app based on the aforementioned signal data, and then rendering is performed.

[0317] While acquiring the rendering resources, you can also obtain the corresponding rendering resource information, which includes, but is not limited to, the number of images N. img Image resolution IW i ×IH i {i=1,2,3,…,N imgNumber of buffer queues N buf The video resolution is BW. j ×BH j {j=1,2,3,…,N buf The display resolution (DW) corresponding to the video. k ×DH k (k = 1, 2, 3, ..., N) buf The corresponding encoding frame rate FR of the video m {m=1,2,3,…,N buf}wait.

[0318] Optionally, during the rendering process, there may be situations where rendering resources are not needed. Therefore, step two is an optional step. In this case, image resources and the buffer queue may not exist, i.e., N. img =0 and N buf =0.

[0319] Step 3: Obtain the desktop image and the corresponding desktop image sequence information.

[0320] Desktop images can be obtained by rendering data once, or by accessing memory; there is no specific limitation on which method is used.

[0321] Desktop image sequence information may include, but is not limited to: the resolution W of the desktop image. s ×H s The refresh rate F of the desktop image sequence ps The encoded frame rate F of the desktop image sequence r Desktop image resolution can be obtained by parsing rendering commands or from system software; there are no specific limitations on this. Desktop image refresh rate can be obtained from rendering services or from system software; there are no specific limitations on this either.

[0322] Step 4: Decision processing mode based on rendering resource information and desktop image sequence information.

[0323] This example provides a decision-making method whose basic criterion is to estimate which processing mode, based on rendered data or based on desktop image sequences, can provide greater benefits, such as producing a lower coding bitrate, lower power consumption, lower complexity, etc., and thus determine the processing mode with greater benefits as the processing mode for the region to be encoded.

[0324] The above decision-making method includes the following five conditions:

[0325] (1) When a buffer queue exists (N) buf >0), based on desktop image resolution W s ×H sand the video resolution BW in the buffer queue j ×BH j {j=1,2,3,…,N buf Decision conditions for}: W s ×H s ≤∑ j (BW j ×BH j )

[0326] That is, the desktop image resolution is less than or equal to the sum of the video resolutions in the buffer queue.

[0327] When this condition is met, the processing mode based on desktop image sequences is expected to be more bitrate-efficient than the processing mode based on rendered data.

[0328] (2) When a buffer queue exists (N) buf >0), based on desktop image resolution W s ×H s The video resolution BW in the buffer queue j ×BH j {j=1,2,3,…,N buf The buffer queue image corresponds to the resolution DW on the display screen. k ×DH k {k = 1, 2, 3, ..., N} buf Decision conditions for}: W s ×H s >∑ j (BW j ×BH j )>∑ k (DW k ×DH k )×R0

[0329] That is, the sum of the video resolutions in the buffer queue is less than the desktop image resolution, and the sum of the video resolutions in the buffer queue is greater than the sum of the product of the resolution of the image in the buffer queue on the display screen and the constant R0, where R0 is a predefined constant, optional, 1≤R0≤2.

[0330] When this condition is met, the processing mode based on desktop image sequences is expected to be more bitrate-efficient than the processing mode based on rendered data.

[0331] (3) When a buffer queue exists (N) buf >0), based on the number of cache queues N buf Decision conditions: N buf >N0

[0332] That is, the number of buffer queues is greater than the constant N0, where N0 is a positive integer, optional, 1≤N0≤4.

[0333] When this condition is met, the processing mode based on rendered data requires encoding multiple videos, thus necessitating the execution of multiple video encoders. This is anticipated to exceed the real-time encoding capabilities of the video encoders, increasing the system's processing burden. In this case, using desktop image encoding requires only a single video encoder, resulting in lower system complexity and lower power consumption.

[0334] (4) When a buffer queue exists (N) buf >0), based on image resolution IW i ×IH i {i=1,2,3,…,N img Desktop image resolution W s ×H s Desktop image display refresh rate F ps Decision conditions: W s ×H s ×F ps / F0≤∑ i (IW i ×IH i )

[0335] That is, the sum of the image resolutions is greater than or equal to the sum of the desktop screen image resolutions within 1 / F0 seconds, where F0 is a positive integer constant, optional, 0 <F0≤10。

[0336] When this condition is met, the processing mode based on rendering data may produce a large instantaneous bitrate. Using the processing mode based on desktop image sequences is expected to save bitrate and provide a smoother, more lag-free display.

[0337] (5) When a buffer queue exists (N) buf >0), based on the corresponding encoded frame rate FR of the video m {m=1,2,3,…,N buf The refresh rate F of the desktop image ps The encoded frame rate F of the desktop image sequence r Decision conditions: F ps <Σ m FR m

[0338] Or, F r <∑ m FR m

[0339] That is, the encoding frame rate (FR) corresponding to the video. m {m=1,2,3,…,N buf The sum of these values ​​is greater than the refresh rate F of the desktop image.ps Or the corresponding video encoding frame rate FR m {m=1,2,3,…,N buf The sum of these values ​​is greater than the encoded frame rate F of the desktop image sequence. r .

[0340] When this condition is met, the first processing mode may generate greater power consumption and bit rate. Using the second processing mode is expected to result in lower power consumption and lower bit rate, thus the second processing mode is more beneficial.

[0341] When any one of the above five conditions is met, the encoding decision unit can decide that the processing mode for the region to be encoded is based on the desktop image sequence. Conversely, when none of the above five conditions are met, the encoding decision unit can decide that the processing mode for the region to be encoded is based on the rendered data.

[0342] Optionally, when there is no buffer queue (N) buf When =0), the encoding decision-maker can decide the processing mode of the region to be encoded as the processing mode based on the rendered data.

[0343] Optionally, when there are no rendering resources (i.e., corresponding to the number N images in the content information), img =0 and the number of videos N buf When =0), the encoding decision-maker can decide the processing mode of the region to be encoded as the processing mode based on the rendered data.

[0344] In this application, the process of obtaining rendering resources required for the rendering process in step two and the process of obtaining desktop images in step three can be executed after the processing mode of the region to be encoded is determined in step four, and there is no specific limitation on the execution order.

[0345] Step 5: Based on the processing mode of the region to be encoded determined by the encoding decision-maker, the content of the region to be encoded is encoded to generate a bitstream.

[0346] If the decision is based on the processing mode of the rendering data, then the content of the region to be encoded (rendering instructions and rendering resources) needs to be encoded. This involves encoding the rendering instructions using a data encoder, encoding the images using an image encoder, and encoding the video in the buffer queue using a video encoder. The encoding methods used by each of the aforementioned encoders are not specifically limited.

[0347] If the decision is based on a desktop image sequence processing mode, then the content of the region to be encoded (rendering instructions and desktop image sequence) needs to be encoded. In this embodiment, rendering instructions still need to be encoded to ensure the continuity of rendering instructions; if the rendering instructions are not continuous, then when switching from a desktop image sequence-based processing mode to a rendering data-based processing mode, more rendering data needs to be sent in order to correctly render the image at the receiving end.

[0348] Optionally, different data type information can be added to the bitstream of the region to be encoded to indicate whether the bitstream it represents is a rendering instruction bitstream, an image bitstream, a video bitstream, etc.

[0349] The bitstream of the region to be encoded can be encapsulated. For example, the different types of bitstreams mentioned above can be spliced ​​together end to end to form a complete frame bitstream.

[0350] Optionally, the processing mode information of the region to be encoded can be added to the bitstream, and the receiving end can obtain the processing mode of the region to be encoded by parsing the bitstream.

[0351] Optionally, the receiving end can be informed of the processing mode for the area to be encoded through a method agreed upon by the projection end and the receiving end. For example, the projection end can use the bitstream of the rendering instructions in the area to be encoded as the last segment of the bitstream in the area to be encoded. When the receiving end parses the bitstream of the rendering instructions in the area to be encoded, it determines whether the preceding bitstream contains an image / video bitstream. If not, the area to be encoded uses a processing mode based on rendering data; if it does, it further determines whether the resolution of the image / video is the same as the resolution of the desktop screen in the area to be encoded. If they are the same, the area to be encoded uses a processing mode based on desktop image sequences; if they are different, the area to be encoded uses a processing mode based on rendering data.

[0352] Figure 7b is an exemplary schematic diagram of the receiving end of this application. As shown in Figure 7b, the receiving end can perform the following steps when reconstructing the entire desktop screen image (the area to be decoded):

[0353] Step 1: Obtain the bitstream of the region to be decoded and decode it according to the bitstream type.

[0354] The receiving end can determine the bitstream type by parsing the data type information in the bitstream of the region to be decoded. For example, it can determine whether the bitstream is a rendering instruction bitstream, an image bitstream, or a video bitstream. Then, the receiving end uses a decoder corresponding to the projection end to decode the different types of bitstreams and reconstruct them into rendering instructions, images, videos, etc.

[0355] Step 2: Determine the processing mode for the region to be decoded.

[0356] This determines whether the projection device uses a processing mode based on rendering data or a processing mode based on desktop image sequences for the area to be decoded.

[0357] Optionally, corresponding to step five on the projection end, if the projection end adds processing mode information to the bitstream, the receiving end can obtain the processing mode of the area to be decoded by parsing the bitstream.

[0358] Optionally, corresponding to step five on the screen-casting end, the receiving end can also determine the processing mode of the area to be decoded through the method agreed upon by the screen-casting end. The method of agreement can be referred to the description in step five on the screen-casting end, and will not be repeated here.

[0359] Step 3: Based on the processing mode of the region to be decoded, process and display the decoded and reconstructed data.

[0360] If the processing mode for the region to be decoded is based on rendering data, the bitstream obtained in step one includes the bitstream of rendering instructions, and may also include the bitstream of images and / or video. Correspondingly, the data for the region to be decoded includes the decoded rendering instructions, and may also include the decoded and reconstructed images and / or video stored in the buffer. The region is rendered based on the rendering instructions and the reconstructed images / video, and then displayed.

[0361] If the processing mode for the region to be decoded is based on a desktop image sequence, the bitstream obtained in step one includes the image bitstream and the rendering instruction bitstream. Correspondingly, the data for the region to be decoded includes the decoded and reconstructed desktop image sequence and the decoded rendering instructions. The reconstructed desktop image sequence is rendered and displayed, while the rendering instructions can be saved or used to build historical rendering information. These rendering instructions can be used to ensure correct rendering when switching to a processing mode based on rendering data later. Alternatively, the reconstructed desktop image sequence can be displayed directly without rendering.

[0362] In this embodiment, the decision-making process for the processing mode combines rendering resource information and desktop image sequence information to determine which processing mode is more beneficial for the area to be decoded. This combines the advantages of the two processing modes, reducing the average bitrate while ensuring the rendering and display effect, thus saving bandwidth resources and improving the user experience.

[0363] Example 3

[0364] The difference between this embodiment and embodiment two is that in embodiment two, the area to be encoded is the entire area of ​​the current desktop, which may include APP, system windows, main interface, etc. In this embodiment, the area to be encoded is the interface of a certain APP or a certain system window. That is, the size of the area to be encoded depends on the size of the APP interface or system window, and can be a part of the current desktop.

[0365] The following steps can be performed on the screen mirroring device:

[0366] Step 1: Obtain the rendering instructions for a specific app / system window.

[0367] The rendering command can be obtained from the rendering service module or from the running program in the aforementioned APP / system window.

[0368] Step Two: The difference between Step Two on the screen projection end and Step Two in Example Two is as follows:

[0369] The rendering process refers to the rendering process of the content within the interface or system window of the aforementioned APP, and the rendering resources also refer to the rendering resources of the content within the interface or system window of the aforementioned APP.

[0370] Step 3: The difference between Step 3 and Step 3 of the screen projection terminal in Example 2 is as follows:

[0371] Obtain the image corresponding to the content within the interface or system window of the aforementioned APP, and obtain relevant information about the image corresponding to the content within the interface or system window of the aforementioned APP.

[0372] Step Four: The difference between Step Four in Example Two and Step Four in the screen projection terminal is as follows:

[0373] The obtained rendering resource information and window image information correspond to the aforementioned APP / system window, and the processing modes are based on rendering data and desktop image sequence.

[0374] Step 5: Same as step 5 of the projection terminal in Example 2.

[0375] The receiving end can perform the following steps:

[0376] Step 1: The difference between Step 1 at the receiving end and Step 2 in Example 2 is as follows:

[0377] The bitstream of the area to be decoded is the bitstream corresponding to the aforementioned APP / system window.

[0378] Step Two: The difference between Step Two at the receiving end and Step Two in Example Two is as follows:

[0379] It determines whether the data to be decoded on the projection end uses a processing mode based on rendering data or a processing mode based on desktop image sequences.

[0380] Step 3: The difference between Step 3 and Step 3 of the receiving end in Embodiment 2 is as follows:

[0381] The data of the area to be decoded can be sent to the graphics rendering service module, or it can be sent to the rendering module corresponding to the APP / system window in the receiving end for rendering processing.

[0382] This embodiment can improve the effect on APP / system windows.

[0383] Example 4

[0384] The difference between Example 4 and Example 2 is that no rendering instructions are encoded / decoded in the desktop image sequence-based processing mode.

[0385] The following steps can be performed on the screen mirroring device:

[0386] Steps 1 to 4 are the same as steps 1 to 4 of the projection end in Example 2.

[0387] Step 5: The difference between Step 5 and Step 5 of the screen projection terminal in Example 2 is as follows:

[0388] If the decision is based on a processing pattern for desktop image sequences, then only the desktop image sequences need to be encoded.

[0389] Optionally, the processing mode information of the region to be encoded can be added to the bitstream, and the receiving end can obtain the processing mode of the region to be encoded by parsing the bitstream.

[0390] Optionally, the processing mode for the region to be encoded can be communicated to the receiving end through a mutually agreed-upon method. For example, in a rendering data-based processing mode, the projection end uses the bitstream of the rendering instructions as the last segment of the bitstream for the region to be encoded. If the receiving end parses the bitstream of the rendering instructions in the region to be encoded, then the region to be encoded uses the rendering data-based processing mode; otherwise, if the resolution of the image / video is the same as the desktop screen resolution, then the region to be encoded uses the desktop image sequence-based processing mode.

[0391] The receiving end can perform the following steps:

[0392] Steps 1 to 2 are the same as steps 1 to 2 of the receiving end in Example 2.

[0393] Step 3: The difference between Step 3 and Step 3 of the receiving end in Embodiment 2 is as follows:

[0394] The description of rendering instructions has been removed in the desktop image sequence-based processing mode.

[0395] If the processing mode for the region to be decoded is based on a desktop image sequence, then the bitstream obtained in step one includes the bitstream of the desktop image. The data for the region to be decoded includes the decoded and reconstructed desktop image.

[0396] In this embodiment, the bitrate is lower in the desktop image sequence-based processing mode. However, when switching from the desktop image sequence-based processing mode to the rendering data-based processing mode, it may be necessary to fully encode and decode the rendering instructions, which will result in a higher bitrate than in Embodiment 2.

[0397] Example 5

[0398] The difference between this embodiment and embodiment two is that the decision-making method of the screen projection terminal for the processing mode is different. This embodiment can make the decision on the processing mode based on information such as APP and system window programs.

[0399] Screen mirroring implementation steps:

[0400] Step 1: Same as Step 1 of the projection terminal in Example 2.

[0401] Step Two: The difference between Step Two on the screen projection end and Step Two in Example Two is as follows:

[0402] You can obtain only the rendering resources required for rendering based on rendering instructions, and you don't need to obtain the corresponding rendering resource information.

[0403] Step 3: The difference between Step 3 and Step 3 of the screen projection terminal in Example 2 is as follows:

[0404] You can acquire only the desktop image, without acquiring the corresponding desktop image sequence information.

[0405] Step 4: Obtain information about the APP / system window program currently displayed on the screen of the projection terminal.

[0406] The above information mainly indicates the APP / system window program that is running and displayed on the screen of the projection terminal, such as the name, identity, and interface display level of the APP / system window program. This information can be obtained from the rendering information of the APP / system window program being rendered received in the rendering service module, or it can be obtained by reading the system task manager; there is no specific limitation on the method of acquisition.

[0407] Step 5: Based on the information from the above APP / system window program, decide on the processing mode for the area to be encoded.

[0408] Based on information about the apps / system windows currently displayed on the projection device's screen, the possible application scenarios of the projection device can be inferred, and the processing mode can be directly determined based on these scenarios. For example, for browser-type apps, it can be inferred that there is a lot of text and images. When such apps are displayed on top, a processing mode based on rendering data can be used. Or, for office software apps, it can be inferred that they mainly contain text and images, without videos, so a processing mode based on rendering data can be used. Or, if only the system desktop is displayed and no other apps are shown, a processing mode based on rendering data can be used. Or, for video playback apps, which may contain a large number of videos, a processing mode based on desktop image sequences can be used when such apps are displayed on top.

[0409] Step Six: Same as Step Five in the projection terminal of Example Two.

[0410] Optionally, steps one through three can be executed after the processing mode is decided in step five. When the decision uses a processing mode based on rendering data, steps one and two are executed, and then step six is ​​executed. When the decision uses a processing mode based on desktop image sequences, steps one (optional) and three are executed, and then step six is ​​executed.

[0411] Example 6

[0412] This embodiment mainly describes the process of switching processing modes.

[0413] The following steps can be performed on the screen mirroring device:

[0414] Step 1: Start the rendering service module of the projection terminal, obtain rendering data, and encode the first frame using the processing mode based on the rendering data.

[0415] In this step, rendering data can be obtained through steps one and two of the projection terminal in Example 2.

[0416] Step 2: Starting from the second frame, for each frame of data, execute steps 1 to 5 of the projection terminal in Example 2. In step 4, when the decision processing mode is the desktop image sequence-based processing mode (different from the processing mode used in the previous frame), the processing mode is switched from the rendering data-based processing mode to the desktop image sequence-based processing mode, and the bitstream in the desktop image sequence-based processing mode is generated according to step 5.

[0417] Step 3: Starting from the second frame, for each frame of data, execute steps 1 to 5 of the projection terminal in Example 2. In step 4, when the decision processing mode is the rendering data-based processing mode (different from the processing mode used in the previous frame), the processing mode is switched from the desktop image sequence-based processing mode to the rendering data-based processing mode, and the bitstream in the rendering data-based processing mode is generated according to step 5.

[0418] The receiving end can perform the following steps:

[0419] Step 1: Start the rendering service module of the receiving end, obtain the bitstream of the first frame, and perform decoding and rendering in accordance with steps 1 to 3 of the receiving end in Embodiment 2 using the processing mode based on rendering data.

[0420] Step 2: Starting from the second frame, for each frame of data, execute steps 1 to 3 of the receiving end in Example 2. When the processing mode is obtained as a desktop image sequence-based processing mode (different from the processing mode used in the previous frame), the rendering service module switches from rendering data processing to rendering desktop image sequence processing.

[0421] Step 3: Starting from the second frame, for each frame of data, execute steps 1 to 3 of the receiving end in Embodiment 2. When the processing mode is obtained as a rendering data-based processing mode (different from the processing mode used in the previous frame), the rendering service module switches from processing the rendering of the desktop image sequence to processing the rendering data.

[0422] Figure 8 is a structural schematic diagram of the hybrid encoding device 800 of this application. As shown in Figure 8, the hybrid encoding device 800 of this embodiment can be applied to the aforementioned projection terminal. The hybrid encoding device 800 may include: an acquisition module 801, a decision module 802, and an encoding module 803.

[0423] The acquisition module 801 is used to acquire content information of the region to be encoded, the content information being used to indicate the content of the region to be encoded; the decision module 802 is used to decide the processing mode of the region to be encoded based on the content information, the processing mode including a first processing mode or a second processing mode; the encoding module 803 is used to encode the content of the region to be encoded according to the processing mode of the region to be encoded to obtain a bitstream.

[0424] In one possible implementation, the content information includes first information and second information, wherein the first information includes resource information of the content of the region to be encoded in the first processing mode, and the second information includes resource information of the content of the region to be encoded in the second processing mode.

[0425] In one possible implementation, the decision module 802 is specifically used to compare the benefits of the first processing mode and the second processing mode based on the first information and the second information; and to determine the processing mode of the region to be encoded as the one with the greater benefit between the first processing mode and the second processing mode.

[0426] In one possible implementation, in the first processing mode, the content of the region to be encoded includes images and / or videos within the region to be encoded. Accordingly, the first information includes information about the images and / or the videos, wherein the image information includes at least one of the number of images or the resolution of the images, and the video information includes at least one of the number of videos, the resolution of the videos, the display resolution corresponding to the videos, or the encoding frame rate of the videos. In the second processing mode, the content of the region to be encoded includes a sequence of desktop images within the region to be encoded. Accordingly, the second information includes at least one of the resolution of the desktop images in the desktop image sequence, the refresh rate of the desktop image sequence, or the encoding frame rate of the desktop image sequence.

[0427] In one possible implementation, the decision module 802 is specifically configured to determine that the benefit of the second processing mode is greater than the benefit of the first processing mode when at least one of the first conditions is met; and to determine that the benefit of the first processing mode is greater than the benefit of the second processing mode when none of the conditions in the first conditions are met; the first conditions include the following:

[0428] When the number of videos is greater than 0, the resolution of the desktop image is less than or equal to the sum of the resolutions of the multiple videos; or,

[0429] When the number of videos is greater than 0, the sum of the resolutions of the multiple videos is less than the resolution of the desktop image, and the sum of the resolutions of the multiple videos is greater than the product of the sum of the display resolutions corresponding to the multiple videos and a first preset value; or,

[0430] The number of videos is greater than a second preset value, where the second preset value is a positive integer; or...

[0431] When the number of videos is greater than 0, the sum of the resolutions of the multiple images is greater than the sum of the resolutions of the desktop images refreshed within a preset time period; or,

[0432] When the number of videos is greater than 0, the sum of the encoded frame rates of the multiple videos is greater than the refresh rate of the desktop image sequence; or,

[0433] When the number of videos is greater than 0, the sum of the encoded frame rates of the multiple videos is greater than the encoded frame rate of the desktop image sequence.

[0434] In one possible implementation, the content information includes third information, which includes attribute information of the content of the region to be encoded, and the attribute information includes at least one of content identifier, content name, or content display level.

[0435] In one possible implementation, the decision module 802 is specifically used to determine the processing mode corresponding to the third information as the processing mode of the region to be encoded based on a pre-set correspondence.

[0436] In one possible implementation, the region to be encoded includes the entire area of ​​the current desktop or the area corresponding to at least one application or window on the current desktop.

[0437] In one possible implementation, the encoding module 803 is specifically used to obtain a rendering instruction when the processing mode of the region to be encoded is the first processing mode, the rendering instruction corresponding to the content of the region to be encoded; and to encode the rendering instruction to obtain the bitstream.

[0438] In one possible implementation, the encoding module 803 is further configured to obtain resources corresponding to the content of the region to be encoded in the first processing mode according to the rendering instructions, the resources including at least one image and / or at least one video; and to encode the resources corresponding to the content of the region to be encoded in the first processing mode to obtain the bitstream.

[0439] In one possible implementation, the encoding module 803 is specifically configured to, when the processing mode of the region to be encoded is the second processing mode, acquire resources corresponding to the content of the region to be encoded in the second processing mode, the resources including desktop image sequences; and encode the resources corresponding to the content of the region to be encoded in the second processing mode to obtain the bitstream.

[0440] In one possible implementation, the encoding module 803 is further configured to acquire rendering instructions, the rendering instructions corresponding to the content of the region to be encoded; and to encode the rendering instructions to obtain the bitstream.

[0441] In one possible implementation, the bitstream further includes processing mode information, which is used to indicate the processing mode of the region to be encoded.

[0442] In one possible implementation, the first processing mode is a processing mode based on rendering data.

[0443] In one possible implementation, the second processing mode is a desktop image sequence-based processing mode.

[0444] In one possible implementation, the bitstream further includes first data type information, which is used to indicate the bitstream corresponding to the rendering instruction.

[0445] In one possible implementation, the bitstream further includes second data type information and / or third data type information, wherein the second data type information is used to indicate the bitstream corresponding to the at least one image, and the third data type information is used to indicate the bitstream corresponding to the at least one video.

[0446] In one possible implementation, the bitstream further includes fourth data type information, which is used to indicate the bitstream corresponding to the desktop image sequence.

[0447] In one possible implementation, the bitstream corresponding to the rendering instruction is located at the end of the bitstream of the region to be encoded.

[0448] In one possible implementation, the bitstream also includes the content information.

[0449] In one possible implementation, the rendering instructions and the at least one image and / or at least one video come from the system's rendering service.

[0450] The apparatus in this embodiment can be used to execute the technical solution of the method embodiment shown in FIG3. Its implementation principle and technical effect are similar, and will not be described again here.

[0451] Figure 9 is a structural schematic diagram of the hybrid decoding device 900 of this application. As shown in Figure 9, the hybrid decoding device 900 of this embodiment can be applied to the receiving end mentioned above. The hybrid decoding device 900 may include: a receiving module 901, a determining module 902, an acquiring module 903, and a rendering module 904.

[0452] The receiving module 901 is used to receive a bitstream; the determining module 902 is used to determine the processing mode of the area to be decoded based on the bitstream, wherein the processing mode includes a first processing mode or a second processing mode; the acquiring module 903 is used to acquire the data corresponding to the content of the area to be decoded based on the bitstream; and the rendering module 904 is used to acquire the image to be displayed based on the processing mode of the area to be decoded and the data corresponding to the content of the area to be decoded.

[0453] In one possible implementation, the bitstream includes processing mode information, which indicates the processing mode of the region to be decoded; correspondingly, the processing mode of the region to be decoded is determined based on the processing mode information.

[0454] In one possible implementation, the determining module 902 is specifically used to obtain content information and data type information of the region to be decoded based on the bitstream. The content information of the region to be decoded is used to indicate the content of the region to be decoded, and the data type information is used to indicate the bitstream of rendering instructions, the bitstream of images, the bitstream of videos, or the bitstream of desktop image sequences. The processing mode of the region to be decoded is determined based on the content information and the data type information.

[0455] In one possible implementation, the data corresponding to the content of the area to be decoded includes rendering instructions; the rendering module 904 is specifically used to render the image to be displayed according to the rendering instructions when the processing mode of the area to be decoded is the first processing mode.

[0456] In one possible implementation, the data corresponding to the content of the area to be decoded includes rendering instructions, and at least one image and / or at least one video; the rendering module 904 is specifically used to render the at least one image and / or the at least one video according to the rendering instructions to obtain the screen to be displayed when the processing mode of the area to be decoded is the first processing mode.

[0457] In one possible implementation, the data corresponding to the content of the area to be decoded includes a desktop image sequence; the rendering module 904 is specifically used to use the desktop image sequence as the screen to be displayed when the processing mode of the area to be decoded is the second processing mode.

[0458] In one possible implementation, the data corresponding to the content of the area to be decoded includes a desktop image sequence and rendering instructions; the rendering module 904 is specifically used to render the desktop image sequence according to the rendering instructions to obtain the screen to be displayed.

[0459] In one possible implementation, the first processing mode is a processing mode based on rendering data.

[0460] In one possible implementation, the second processing mode is a desktop image sequence-based processing mode.

[0461] The apparatus in this embodiment can be used to execute the technical solution of the method embodiment shown in FIG5. Its implementation principle and technical effect are similar, and will not be described again here.

[0462] In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware encoding processor, or by a combination of hardware and software modules in the encoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0463] The memory mentioned in the above embodiments can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0464] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0465] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0466] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0467] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0468] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0469] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0470] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A hybrid encoding method, characterized in that, include: Obtain content information of the region to be encoded, wherein the content information is used to indicate the content of the region to be encoded; The processing mode of the region to be encoded is determined based on the content information, and the processing mode includes a first processing mode or a second processing mode. The content of the region to be encoded is encoded according to the processing mode of the region to be encoded to obtain a bitstream.

2. The method according to claim 1, characterized in that, The content information includes first information and second information. The first information includes resource information of the content of the region to be encoded in the first processing mode, and the second information includes resource information of the content of the region to be encoded in the second processing mode.

3. The method according to claim 2, characterized in that, The step of determining the processing mode of the region to be encoded based on the content information includes: Compare the benefits of the first processing mode and the second processing mode based on the first information and the second information; The processing mode with the greater benefit between the first processing mode and the second processing mode is determined as the processing mode for the region to be encoded.

4. The method according to claim 2 or 3, characterized in that, In the first processing mode, the content of the region to be encoded includes images and / or videos within the region to be encoded. Accordingly, the first information includes information about the images and / or information about the videos. The information about the images includes at least one of the number of images or the resolution of the images. The information about the videos includes at least one of the number of videos, the resolution of the videos, the display resolution corresponding to the videos, or the encoding frame rate of the videos. In the second processing mode, the content of the region to be encoded includes a sequence of desktop images of the region to be encoded. Accordingly, the second information includes at least one of the resolution of the desktop images in the desktop image sequence, the refresh rate of the desktop image sequence, or the encoding frame rate of the desktop image sequence.

5. The method according to claim 4, characterized in that, The step of comparing the benefits of the first processing mode and the second processing mode based on the first information and the second information includes: When at least one of the first conditions is met, it is determined that the benefit of the second processing mode is greater than the benefit of the first processing mode. If none of the conditions in the first condition are met, it is determined that the benefit of the first processing mode is greater than the benefit of the second processing mode. The first condition includes the following conditions: When the number of videos is greater than 0, the resolution of the desktop image is less than or equal to the sum of the resolutions of the multiple videos; or, When the number of videos is greater than 0, the sum of the resolutions of the multiple videos is less than the resolution of the desktop image, and the sum of the resolutions of the multiple videos is greater than the product of the sum of the display resolutions corresponding to the multiple videos and a first preset value; or, The number of videos is greater than a second preset value, where the second preset value is a positive integer; or... When the number of videos is greater than 0, the sum of the resolutions of the multiple images is greater than the sum of the resolutions of the desktop images refreshed within a preset time period; or, When the number of videos is greater than 0, the sum of the encoded frame rates of the multiple videos is greater than the refresh rate of the desktop image sequence; or, When the number of videos is greater than 0, the sum of the encoded frame rates of the multiple videos is greater than the encoded frame rate of the desktop image sequence.

6. The method according to claim 1, characterized in that, The content information includes third information, which includes attribute information of the content of the area to be encoded. The attribute information includes at least one of the following: content identifier, content name, or content display level.

7. The method according to claim 6, characterized in that, The step of determining the processing mode of the region to be encoded based on the content information includes: Based on a pre-defined correspondence, the processing mode corresponding to the third information is determined as the processing mode for the region to be encoded.

8. The method according to any one of claims 1-7, characterized in that, The area to be encoded includes the entire area of ​​the current desktop or the area corresponding to at least one application or window on the current desktop.

9. The method according to any one of claims 1-8, characterized in that, The step of encoding the content of the region to be encoded according to the processing mode of the region to be encoded to obtain a bitstream includes: When the processing mode of the region to be encoded is the first processing mode, a rendering instruction is obtained, and the rendering instruction corresponds to the content of the region to be encoded. The rendering instructions are encoded to obtain the bitstream.

10. The method according to claim 9, characterized in that, The step of encoding the content of the region to be encoded according to the processing mode of the region to be encoded to obtain a bitstream further includes: According to the rendering instructions, resources corresponding to the content of the region to be encoded in the first processing mode are obtained, and the resources include at least one image and / or at least one video. The resources corresponding to the content of the region to be encoded in the first processing mode are encoded to obtain the bitstream.

11. The method according to any one of claims 1-8, characterized in that, The step of encoding the content of the region to be encoded according to the processing mode of the region to be encoded to obtain a bitstream includes: When the processing mode of the region to be encoded is the second processing mode, resources corresponding to the content of the region to be encoded under the second processing mode are obtained, and the resources include desktop image sequences; The resources corresponding to the content of the region to be encoded in the second processing mode are encoded to obtain the bitstream.

12. The method according to claim 11, characterized in that, The step of encoding the content of the region to be encoded according to the processing mode of the region to be encoded to obtain a bitstream further includes: Obtain rendering instructions, wherein the rendering instructions correspond to the content of the region to be encoded; The rendering instructions are encoded to obtain the bitstream.

13. The method according to any one of claims 1-12, characterized in that, The bitstream also includes processing mode information, which is used to indicate the processing mode of the region to be encoded.

14. The method according to any one of claims 1-13, characterized in that, The first processing mode is a processing mode based on rendering data.

15. The method according to any one of claims 1-14, characterized in that, The second processing mode is a desktop image sequence-based processing mode.

16. The method according to claim 9 or 12, characterized in that, The bitstream also includes first data type information, which is used to indicate the bitstream corresponding to the rendering instruction.

17. The method according to claim 10, characterized in that, The bitstream further includes second data type information and / or third data type information, wherein the second data type information is used to indicate the bitstream corresponding to the at least one image, and the third data type information is used to indicate the bitstream corresponding to the at least one video.

18. The method according to claim 11, characterized in that, The bitstream also includes fourth data type information, which is used to indicate the bitstream corresponding to the desktop image sequence.

19. The method according to claim 9 or 12, characterized in that, The bitstream corresponding to the rendering instruction is located at the end of the bitstream of the region to be encoded.

20. The method according to any one of claims 1-19, characterized in that, The bitstream also includes the content information.

21. The method according to claim 10, characterized in that, The rendering instructions and the at least one image and / or at least one video come from the system's rendering service.

22. A hybrid decoding method, characterized in that, include: Receive bitstream; The processing mode of the region to be decoded is determined based on the bitstream, and the processing mode includes a first processing mode or a second processing mode. The data corresponding to the content of the region to be decoded is obtained based on the bitstream; The image to be displayed is obtained based on the processing mode of the area to be decoded and the data corresponding to the content of the area to be decoded.

23. The method according to claim 22, characterized in that, The bitstream includes processing mode information, which is used to indicate the processing mode of the region to be decoded; correspondingly, the processing mode of the region to be decoded is determined based on the processing mode information.

24. The method according to claim 22, characterized in that, The process of determining the processing mode of the region to be decoded based on the bitstream includes: The content information and data type information of the region to be decoded are obtained according to the bitstream. The content information of the region to be decoded is used to indicate the content of the region to be decoded, and the data type information is used to indicate the bitstream of rendering instructions, the bitstream of images, the bitstream of videos, or the bitstream of desktop image sequences. The processing mode of the region to be decoded is determined based on the content information and the data type information of the region to be decoded.

25. The method according to any one of claims 22-24, characterized in that, The data corresponding to the content of the area to be decoded includes rendering instructions; obtaining the image to be displayed based on the processing mode of the area to be decoded and the data corresponding to the content of the area to be decoded includes: When the processing mode of the area to be decoded is the first processing mode, the image to be displayed is rendered according to the rendering instruction.

26. The method according to any one of claims 22-24, characterized in that, The data corresponding to the content of the area to be decoded includes rendering instructions, and at least one image and / or at least one video; obtaining the screen to be displayed based on the processing mode of the area to be decoded and the data corresponding to the content of the area to be decoded includes: When the processing mode of the area to be decoded is the first processing mode, the at least one image and / or the at least one video are rendered according to the rendering instructions to obtain the screen to be displayed.

27. The method according to any one of claims 22-24, characterized in that, The data corresponding to the content of the area to be decoded includes a desktop image sequence; obtaining the screen to be displayed based on the processing mode of the area to be decoded and the data corresponding to the content of the area to be decoded includes: When the processing mode of the area to be decoded is the second processing mode, the desktop image sequence is used as the screen to be displayed.

28. The method according to any one of claims 22-24, characterized in that, The data corresponding to the content of the area to be decoded includes a desktop image sequence and rendering instructions; obtaining the screen to be displayed based on the processing mode of the area to be decoded and the data corresponding to the content of the area to be decoded includes: The desktop image sequence is rendered according to the rendering instructions to obtain the screen to be displayed.

29. The method according to any one of claims 22-28, characterized in that, The first processing mode is a processing mode based on rendering data.

30. The method according to any one of claims 22-29, characterized in that, The second processing mode is a desktop image sequence-based processing mode.

31. A bitstream, characterized in that, The bitstream includes a first bitstream obtained based on a first processing mode and a second bitstream obtained based on a second processing mode.

32. The bitstream according to claim 31, characterized in that, The first processing mode is a processing mode based on rendering data, and the second processing mode is a processing mode based on desktop image sequences.

33. A terminal device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-30.

34. A computer-readable storage medium, characterized in that, Includes a computer program, which, when executed on a computer, causes the computer to perform the method of any one of claims 1-30.

35. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to perform the method of any one of claims 1-30.

Images