Data processing method, system and electronic device

Abstract

This application provides a data processing method, system, and electronic device. The method includes: before the encoder encodes an image rendered by a graphics processor, the graphics processor acquires rendering information related to the rendered image, then compresses and transmits the compressed rendering information; subsequently, during the encoder's encoding of the rendered image, it can perform encoding optimization based on the compressed rendering information. This reduces the amount of rendering information, thereby reducing the transmission latency of the rendering information. Furthermore, the encoder performs encoding only after receiving the rendering information; therefore, this application can reduce the encoder's waiting time for encoding, thereby reducing the overall transmission latency of the cloud rendering process.

Description

[0001] This application is a divisional application. The original application has the application number 202110786721.X and the original application date is July 12, 2021. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of data processing, and more particularly to a data processing method, system, and electronic device. Background Technology

[0003] Currently, many special effects in games, animations, and movies / TV are achieved through rendering. Because rendering is computationally complex, it places extremely high demands on the computing power of devices. This has led to the development of cloud rendering, which places the rendering operation on a server to reduce the computational demands on the client.

[0004] However, cloud rendering has high requirements for data transmission bandwidth and latency. Therefore, how to reduce data transmission bandwidth requirements or latency has become an urgent problem to be solved. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a data processing method, system, and electronic device. In this method, rendering information is compressed before transmission, and / or the rendering information is transmitted in chunks, reducing data transmission bandwidth requirements and transmission latency.

[0006] Firstly, embodiments of this application provide a data processing method, comprising: before an encoder encodes an image rendered by a graphics processor, the graphics processor acquires rendering information related to the rendered image, then compresses the rendering information and transmits the compressed rendering information; subsequently, during the encoding process of the rendered image, the encoder can perform encoding optimization based on the compressed rendering information. This reduces the amount of rendering information data, thereby reducing the transmission latency of the rendering information. Furthermore, since the encoder performs encoding only after receiving the rendering information, embodiments of this application can reduce the encoder's waiting time for encoding, thereby reducing the transmission latency of the entire cloud rendering process. In addition, the reduction in the amount of rendering information data can also reduce the bandwidth requirements of the rendering information transmission. Moreover, the cache space used to store rendering information in the encoder can also be reduced, thereby saving encoder cache space.

[0007] For example, rendering information may include information used to assist in the execution of the rendering process.

[0008] For example, rendering information may include information input to the rendering module, such as depth information and material ID (Identity Document) information (hereinafter referred to as ID information).

[0009] For example, rendering information may also include information generated by the rendering module during the rendering process, such as RMV (Rendering Motion Vector) information.

[0010] According to the first aspect, compressing the rendering information includes: reducing the resolution of the rendering information; and / or, reducing the bit depth of the rendering information.

[0011] According to the first aspect, or any implementation of the first aspect above, reducing the resolution of the rendering information includes: obtaining sampling ratio information, which includes horizontal sampling ratio information and vertical sampling ratio information; and downsampling the rendering information in the horizontal and vertical dimensions based on the horizontal sampling ratio information and the vertical sampling ratio information.

[0012] According to the first aspect, or any implementation of the first aspect above, the sampling ratio information is obtained, including: obtaining the storage granularity information of the encoder for the motion vector, and determining the sampling ratio information based on the storage granularity information; or, obtaining the compensation granularity information required for motion compensation by the decoder, and determining the sampling ratio information based on the compensation granularity information.

[0013] According to the first aspect, or any implementation of the first aspect above, the bit depth of the rendering information is a first bit depth value; reducing the bit depth of the rendering information includes: obtaining a second bit depth value, the second bit depth value being lower than the first bit depth value; and converting the bit depth of the rendering information from the first bit depth value to the second bit depth value.

[0014] According to the first aspect, or any implementation of the first aspect above, obtaining the second bit depth value includes: obtaining a third bit depth value, the third bit depth value being used to characterize the bit depth of the rendering information required for encoder encoding optimization; and determining the second bit depth value based on the first bit depth value and the third bit depth value.

[0015] According to the first aspect, or any implementation thereof, transmitting compressed rendering information includes: dividing the compressed rendering information into multiple information blocks; and transmitting each information block separately. In this way, when the encoder receives any information block of the compressed rendering information, it can encode the macroblock corresponding to that information block; thereby enabling the transmission of rendering information and encoding to proceed synchronously, reducing the encoder's waiting time for rendering information, further reducing the transmission latency of the rendering information, and also further reducing the bandwidth requirements of the rendering information.

[0016] According to the first aspect, or any implementation of the first aspect above, the method further includes: dividing the rendering information into multiple information blocks; compressing the rendering information including: compressing each information block separately; and transmitting the compressed rendering information including: transmitting each compressed information block separately. In this way, when the encoder receives any information block of the compressed rendering information, it can encode the macroblock corresponding to that information block; thereby enabling the transmission of rendering information and encoding to proceed synchronously, reducing the encoder's waiting time for rendering information, further reducing the transmission latency of the rendering information, and further reducing the bandwidth requirements of the rendering information.

[0017] According to the first aspect, or any implementation of the first aspect above, the compressed rendering information is divided into multiple information blocks, including: dividing the compressed rendering information into blocks according to a preset block division method to obtain multiple information blocks; wherein, the block division method is the way the encoder divides the rendered image into multiple macroblocks.

[0018] According to the first aspect, or any implementation of the first aspect above, each information block is transmitted separately, including: determining the transmission order of each information block according to a preset encoding order, wherein the encoding order is the encoding order of the encoder for multiple macroblocks; and transmitting each information block sequentially according to the transmission order corresponding to each information block.

[0019] According to the first aspect, or any implementation of the first aspect above, transmitting compressed rendering information includes: transmitting compressed rendering information to an encoder; or, transmitting compressed rendering information to an analysis module, so that the analysis module can analyze the compressed rendering information, determine encoding optimization information, and transmit the encoding optimization information to the encoder, so that the encoder can perform encoding optimization based on the encoding optimization information during the encoding process of the rendered image.

[0020] Secondly, embodiments of this application provide a data processing method, comprising: before the encoder encodes an image rendered by a graphics processor, the graphics processor acquires rendering information related to the rendered image, then divides the rendering information into multiple information blocks, and transmits each information block separately. In this way, when the encoder receives any information block of the rendering information, it can encode the macroblock corresponding to that information block; thereby enabling the transmission of rendering information and encoding to proceed synchronously, reducing the encoder's waiting time for rendering information, and thus reducing the transmission latency of the rendering information. Furthermore, the data size of an information block is smaller than the data size of the complete rendering information; therefore, embodiments of this application can also reduce the bandwidth requirements for the transmission of rendering information.

[0021] According to the second aspect, the rendering information is divided into multiple information blocks, including: dividing the rendering information into blocks according to a preset block division method to obtain multiple information blocks; wherein, the block division method is the way the encoder divides the rendered image into multiple macroblocks.

[0022] According to the second aspect, or any implementation of the second aspect above, each information block is transmitted separately, including: determining the transmission order of each information block according to a preset encoding order, wherein the encoding order is the encoding order of the encoder for multiple macroblocks; and transmitting each information block sequentially according to the transmission order corresponding to each information block.

[0023] For example, after the encoder completes the encoding optimization using an information block, it can release the cache corresponding to that information block to save the encoder's cache space.

[0024] According to the second aspect, or any implementation of the second aspect above, each information block is transmitted separately, including: transmitting each information block to the encoder separately; or, transmitting each information block to the analysis module separately, so that the analysis module analyzes based on each information block, determines the encoding optimization information corresponding to each information block, and transmits the encoding optimization information to the encoder, so that the encoder performs encoding optimization based on the encoding optimization information of the corresponding information block during the encoding process of each macroblock.

[0025] Thirdly, embodiments of this application provide a graphics processor for executing the data processing method in the first aspect or any possible implementation of the first aspect or the second aspect or any possible implementation of the second aspect.

[0026] The third aspect and any implementation thereof correspond to the first aspect, any implementation thereof, the second aspect, and any implementation thereof, respectively. The technical effects of the third aspect and any implementation thereof can be found in the technical effects of the first aspect, any implementation thereof, the second aspect, and any implementation thereof, as described above, and will not be repeated here.

[0027] Fourthly, embodiments of this application provide a data processing system, including: a graphics processor and an encoder; the graphics processor is used for image rendering and acquiring rendering information related to the rendered image; transmitting the rendered image to the encoder, and compressing the rendering information and transmitting the compressed rendering information to the encoder; the encoder is used for encoding the rendered image and optimizing the encoding based on the compressed rendering information during the encoding process.

[0028] The fourth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the fourth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0029] Fifthly, embodiments of this application provide a data processing system, including: a graphics processor and an encoder; the graphics processor is used for image rendering and acquiring rendering information related to the rendered image; transmitting the rendered image to the encoder, and dividing the rendering information into multiple information blocks, and transmitting each information block to the encoder respectively; the encoder is used for dividing the rendered image into multiple macroblocks, and after receiving an information block, encoding the macroblock corresponding to the received information block, and performing encoding optimization based on the received information block during the encoding process of the macroblock.

[0030] The fifth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the fifth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0031] In a sixth aspect, embodiments of this application provide a data processing system, including: a graphics processor, an analysis module, and an encoder. The graphics processor is used for image rendering and acquiring rendering information related to the rendered image; transmitting the rendered image to the encoder, compressing the rendering information, and transmitting the compressed rendering information to the analysis module; the analysis module is used for analyzing the compressed rendering information, determining encoding optimization information, and transmitting the encoding optimization information to the encoder; the encoder is used for encoding the rendered image and performing encoding optimization based on the encoding optimization information during the encoding process.

[0032] The sixth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the sixth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0033] In a seventh aspect, embodiments of this application provide a data processing system, including: a graphics processor, an analysis module, and an encoder. The graphics processor is used for image rendering and acquiring rendering information related to the rendered image; transmitting the rendered image to the encoder; and dividing the rendering information into multiple information blocks and transmitting each information block to the analysis module. The analysis module is used for analyzing each information block, determining the encoding optimization information corresponding to each information block, and transmitting the encoding optimization information corresponding to each information block to the encoder. The encoder is used for dividing the rendered image into multiple macroblocks, and after receiving an information block, encoding the macroblock corresponding to the received information block, and performing encoding optimization based on the encoding optimization information of the received information block during the encoding process of the macroblock.

[0034] The seventh aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the seventh aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0035] Eighthly, embodiments of this application provide an electronic device, including: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, cause the electronic device to perform a data processing method in the third aspect or any possible implementation thereof.

[0036] The eighth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the eighth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0037] Ninthly, embodiments of this application provide a computer storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to perform the data processing method in the third aspect or any possible implementation thereof.

[0038] The ninth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects corresponding to the ninth aspect and any implementation thereof are similar to those corresponding to the first aspect and any implementation thereof, and will not be repeated here. Attached Figure Description

[0039] Figure 1 This is a schematic diagram illustrating an application scenario.

[0040] Figure 2 This is a schematic diagram illustrating the image segmentation process as an example.

[0041] Figure 3 This is a schematic diagram illustrating data transmission as an example.

[0042] Figure 4 This is a schematic diagram illustrating data transmission as an example.

[0043] Figure 5 This is a schematic diagram illustrating a data processing procedure as an example.

[0044] Figure 6 This is a schematic diagram illustrating a data transmission process as an example.

[0045] Figure 7 This is a schematic diagram illustrating data processing as an example.

[0046] Figure 8 This is a schematic diagram illustrating a data transmission process as an example.

[0047] Figure 9 This is a schematic diagram illustrating a data processing procedure as an example.

[0048] Figure 10 This is a schematic diagram of a block structure as an example.

[0049] Figure 11 This is a schematic diagram illustrating the information block transmission process as an example.

[0050] Figure 12 This is an illustrative diagram of transmission and encoding.

[0051] Figure 13 This is an illustrative diagram of transmission and encoding.

[0052] Figure 14 This is a schematic diagram illustrating a data processing procedure as an example.

[0053] Figure 15 This is a schematic diagram illustrating a data transmission process as an example.

[0054] Figure 16 This is a schematic diagram illustrating a data processing procedure as an example.

[0055] Figure 17 This is a schematic diagram illustrating a data transmission process as an example.

[0056] Figure 18 This is a schematic diagram illustrating a data processing procedure as an example.

[0057] Figure 19 This is a schematic diagram illustrating a data transmission process as an example.

[0058] Figure 20 This is a schematic diagram illustrating a data transmission process as an example.

[0059] Figure 21 This is a schematic diagram illustrating a data processing procedure as an example.

[0060] Figure 22 This is a schematic diagram illustrating a data transmission process as an example.

[0061] Figure 23 This is a schematic diagram illustrating a data transmission process as an example.

[0062] Figure 24 This is a schematic diagram of the structure of an exemplary device. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0064] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0065] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0066] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0067] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0068] This application's embodiments can be applied to scenarios requiring cloud rendering, such as AR (Augmented Reality) scenarios, VR (Virtual Reality) scenarios, cloud exhibition scenarios, cloud classroom scenarios, cloud karaoke scenarios, cloud concert scenarios, cloud gaming scenarios, and content production scenarios (including film special effects production, television feature production, etc.). This application's embodiments do not impose any limitations on these scenarios. This application's embodiments use a cloud gaming scenario as an example for illustrative explanation.

[0069] Figure 1 This is a schematic diagram illustrating an exemplary application scenario. (Refer to...) Figure 1 , Figure 1 The application scenario in this context is cloud gaming.

[0070] Figure 1 The cloud gaming scenario includes: client and server.

[0071] For example, the client deploys a game application and a decoding module. For example, the game application can provide a user interface for game operations. For example, the decoding module can be used to decode data (such as images / videos) and can be a decoder (such as a video decoder).

[0072] It should be understood that the client has more than Figure 1 The embodiments of this application do not limit the scope of the additional modules or programs shown.

[0073] For example, the server is deployed with: a parsing module, a game logic processing module, a rendering module, an encoding module, and a transmission module.

[0074] For example, the parsing module can be used to parse the indication information.

[0075] For example, the game logic processing module can be used to determine whether rendering is required.

[0076] For example, a rendering module can be used to perform rendering operations.

[0077] For example, the encoding module can be used to encode data (such as images / videos) and can be an encoder (such as a video encoder).

[0078] For example, the transmission module can be used to transmit encoded data (such as compressed image / video data).

[0079] For example, the parsing module, game logic processing module, encoding module, and transmission module can be deployed in a CPU (Central Processing Unit). The rendering module can be a graphics processing unit (GPU), or it can be deployed in a GPU. For example, a graphics processing unit is a microprocessor specifically designed for performing image and graphics-related computations on personal computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones). A graphics processing unit can include a GPU.

[0080] For example, the parsing module, game logic processing module, and transmission module can be deployed on the CPU, while the rendering module and encoding module can be deployed on the GPU.

[0081] It should be understood that servers have more than Figure 1 The embodiments of this application do not limit the scope of the additional modules shown.

[0082] Continue to refer to Figure 1 The rendering process for cloud gaming can be described as follows: S101, the game application receives user input.

[0083] For example, a user can open the game application in the client as needed and perform various operations within the game application to play the game. The game application can then receive user input.

[0084] S102, the game application sends an instruction message to the parsing module.

[0085] For example, after receiving a user's operation, the game application can respond to the user's operation by generating instruction information based on information related to the user's operation, and then send the instruction information to the server's parsing module.

[0086] S103, the parsing module sends game instructions to the game logic processing module.

[0087] For example, after receiving the instruction information, the parsing module can parse the instruction information to obtain the game command. Then, the parsing module sends the game command to the game logic processing module.

[0088] S104, the game logic processing module sends rendering instructions to the rendering module.

[0089] For example, after receiving a game instruction, the game logic processing module can determine whether rendering is required based on the instruction. If rendering is determined to be required, a rendering instruction can be generated and sent to the rendering module.

[0090] S105, the rendering module sends the rendered image and rendering information to the encoding module.

[0091] For example, after receiving a rendering instruction, the rendering module can load the data required for rendering based on the rendering instruction. For example, the data required for rendering may include the model data of the 3D model corresponding to the rendering instruction. The model data of the 3D model includes, but is not limited to, vertex coordinates, texture coordinates, texture maps, etc., and this application embodiment does not limit this.

[0092] For example, after the rendering module loads the data required for rendering, it can render based on the data required for rendering to obtain the image of the current game scene, that is, the rendered image.

[0093] For example, the rendering module can send the rendered image to the encoding module, which encodes the rendered image before transmitting it to the client.

[0094] For example, the rendering module can also send rendering information to the encoding module so that the encoding module can optimize the encoding based on the rendering information and improve the encoding efficiency.

[0095] For example, rendering information may include information used to assist in the execution of the rendering process. For example, rendering information may include information input to the rendering module, such as depth information and material ID (Identity Document) information (hereinafter referred to as ID information). For example, rendering information may also include information generated by the rendering module during the rendering process, such as RMV (Rendering Motion Vector) information.

[0096] For example, the ID information includes pixel information of each pixel after mapping each vertex of an object in a 3D model from 3D (3-Dimensional) space to a pixel on the screen.

[0097] For example, the bit depth of ID information is K1 bits (K1 is a positive integer, such as 8). The bit depth of information can refer to the number of bits corresponding to the value of the information. ID information can include 3 channels, and the data in all 3 channels is valid data. The value of each channel can be represented by K1 bits. For example, ID information can be represented by three channels: RGB (R (Red), G (Green), and B (Blue)). The ID value of the same object or the same material is the same (all three RGB values ​​are the same).

[0098] For example, depth information includes the depth value of each pixel after each vertex of an object in a 3D model is mapped from 3D space to a pixel on the screen.

[0099] For example, the bit depth of the depth information is K2 bits (K2 is a positive integer, such as 8), and it can include 3 channels. The data of one of the 3 channels is valid data, and the value of each channel can be represented by K2 bits.

[0100] For example, RMV information may include the position difference between each vertex of the depth-tested 3D model and the pixel position mapped to the screen in the m-th (m is a positive integer) frame and the pixel position mapped to the screen in the (m+1)-th frame.

[0101] For example, the bit depth of RMV information is K3 bits (K3 is a positive integer, such as 32), and it can include 3 channels. The data in 2 of the 3 channels is valid data, and the value of each channel can be represented by a floating-point number of K3 bits.

[0102] For example, the rendering information has the same resolution as the rendered image. For instance, if the resolution of the rendered image is W×H, then the resolution of the rendering information can be W×H. Here, W represents horizontal pixels and H represents vertical pixels.

[0103] It should be noted that the rendering information in this embodiment contains more information than that shown above, such as camera parameters (including but not limited to: focal length, aperture, camera intrinsic and extrinsic parameters, exposure, etc.), and this embodiment does not limit this.

[0104] For example, the rendering module may transmit some rendering information to the encoding module. For instance, the rendering module may transmit one of the ID information, Depth information, and RMV information to the encoding module; alternatively, the rendering module may transmit two of the ID information, Depth information, and RMV information to the encoding module; or, for yet another example, the rendering module may transmit all of the ID information, Depth information, and RMV information to the encoding module. For example, the rendering module may also transmit all of the rendering information to the encoding module; this embodiment of the application does not limit this.

[0105] S106, the encoding module sends the encoded data to the transmission module.

[0106] After receiving the rendered image and rendering information, the encoding module can encode the rendered image and optimize the encoding based on the rendering information during the encoding process. Once the rendered image has been encoded to obtain the corresponding encoded data, the encoded data can be sent to the transmission module.

[0107] For example, before rendering the rendered image, the encoding module can first divide the rendered image into multiple macroblocks, then divide each macroblock into multiple encoding blocks, and then encode the image in units of encoding blocks.

[0108] Figure 2 This is a schematic diagram illustrating the image segmentation process as an example.

[0109] Reference Figure 2 (1), Figure 2 (1) is a frame of rendered image with a resolution of 64×64. Then, the rendered image is divided into multiple macroblocks, such as... Figure 2 As shown in (2), the rendered image can be divided into four macroblocks: macroblock 1, macroblock 2, macroblock 3, and macroblock 4. The resolution of macroblocks 1, 2, 3, and 4 is 32×32. Each macroblock is then further divided into multiple coding blocks, such as... Figure 2 As shown in (3), macroblock 4 can be divided into 16 coding blocks. Among them, coding blocks 8 and 16 have a resolution of 16×16, coding blocks 1, 2, 7, 13, 14 and 15 have a resolution of 8×8, and coding blocks 3, 4, 5, 6, 9, 10, 11 and 12 have a resolution of 4×4.

[0110] It should be noted that, Figure 2 (3) is just one example of dividing a macroblock into multiple coding blocks. Macroblock 4 can also be divided into coding blocks with smaller granularity, such as 2×2 coding blocks, etc. This application does not limit this.

[0111] It should also be noted that macroblock 1, macroblock 2 and macroblock 3 can be divided into coding blocks in the same way as macroblock 4, or they can be divided into coding blocks in a different way than macroblock 4. This application embodiment does not limit this.

[0112] For example, the encoding process of the encoding module is as follows: For each encoding block, the pixel values ​​of pixels within the reconstructed region of the current frame can be used to predict the pixel values ​​of pixels within the current encoding block (i.e., intra-frame prediction). Alternatively, in the reconstructed image frame, a matching reference encoding block is found for the current encoding block in the current frame, and the pixel values ​​of pixels in the reference encoding block are used as the prediction information for the pixel values ​​of pixels in the current encoding block (this process can be called motion estimation (ME)); and the motion vector pointing to the reference encoding block (MV) and the indication information of the image frame where the reference encoding block is located are determined (i.e., inter-frame prediction). After obtaining the prediction information through intra-frame or inter-frame prediction, the corresponding prediction information can be subtracted from the pixel values ​​of pixels within the current encoding block to obtain residual information. Then, methods such as Discrete Cosine Transform (DCT) are used to transform the residual information, and quantization entropy coding is used to complete the encoding of a encoding block, thus obtaining the encoded data of a encoding block.

[0113] For example, the encoding module can optimize encoding based on rendering information in various ways. For instance, the encoding module can use the RMV information corresponding to the current encoding block as the search starting point for the MV or motion estimation of the current encoding block to accelerate the motion estimation process. Another example is that the encoding module can determine the direction of the texture based on ID information, Depth information, or RMV information, and then determine the partitioning mode of the encoding block (for example, if the Depth information (or ID information or RMV information) corresponding to each pixel in the current encoding block is the same or similar (for example, the transformation range of the Depth information (or ID information or RMV information) is less than a predetermined threshold), then it is considered to be the same object. In this case, the current encoding block can be considered to no longer be further partitioned to save encoding time).

[0114] S107, the transmission module sends the encoded data to the decoding module.

[0115] For example, the transmission module performs streaming processing on the encoded data and transmits the streamed data.

[0116] For example, the transmission module can obtain indication information from the parsing module, parse the indication information, and obtain the source address of the client.

[0117] For example, the transmission module can also obtain the client's source address directly from the parsing module.

[0118] Then the transmission module can send the encoded data to the corresponding client's decoding module based on the source address.

[0119] S108, the decoding module sends the decoded rendered image to the game application.

[0120] After receiving the encoded data, the decoding module can decode the encoded data to obtain the decoded rendered image, and then return the decoded rendered image to the game application.

[0121] It should be noted that because the encoding module encodes (i.e. compresses) the rendered image, the image quality will decrease. Therefore, the decoded rendered image sent by the decoding module to the game application is different from the rendered image sent by the rendering module to the encoding module.

[0122] For example, the decoding process of the decoding module can be as follows: The decoding module first obtains residual information through entropy decoding, inverse quantization, and inverse transform. Then, it determines whether the current coding block uses intra-frame predictive coding or inter-frame predictive coding. If it is intra-frame predictive coding, the prediction information is constructed using the pixel values ​​of the surrounding reconstructed pixels according to the intra-frame prediction method used by the coding module. If it is inter-frame predictive coding, the motion information MV needs to be parsed, and the parsed motion information MV is used to determine the reference coding block in the reconstructed image. The pixel values ​​of the pixels within the block are used as the prediction information (this process is called motion compensation (MC)). The reconstructed information can be obtained by using the prediction information plus the residual information and performing a filtering operation.

[0123] After receiving the decoded rendered image returned by the decoding module, the game application can display the decoded rendered image; thereby displaying the game scene (image or video) corresponding to the user's operation.

[0124] Figure 1 S105 involves a large amount of data transmission, which makes the transmission bandwidth and latency requirements of this process relatively high. Therefore, it is possible to consider reducing the transmission bandwidth requirements or transmission latency. Thus, this embodiment of the application starts with S105 to reduce the transmission bandwidth requirements and transmission latency.

[0125] In existing technologies, the rendering module only begins encoding the rendered image after receiving the rendering information of each frame of the rendered image, and the rendering information is only used for encoding optimization. Furthermore, the rendering information includes various types, and some of the rendering information (e.g., depth information, ID information, RMV information) has the same resolution as the rendered image. In other words, the amount of data in the rendering information is many times that of the rendered image. Therefore, in this embodiment, we can consider starting with the rendering information to reduce the transmission bandwidth requirements and transmission latency.

[0126] One possible implementation is to first compress the rendering information to reduce the amount of data; then transmit the compressed rendering information, which can reduce the transmission latency of the rendering information and also reduce the bandwidth requirements of the rendering information.

[0127] For example, this application embodiment uses a GPU as the rendering module, an encoder as the encoding module, and a decoder as the decoding module for illustrative purposes.

[0128] Figure 3 This is a schematic diagram illustrating data transmission as an example.

[0129] Reference Figure 3 For example, the GPU transmits the compressed rendering information to the analysis module. The analysis module analyzes the compressed rendering information to determine the encoding optimization information needed by the encoder. Then, the analysis module sends the encoding optimization information to the encoder, which performs encoding optimization based on this information during the encoding process of the rendered image.

[0130] For example, the analysis module can be deployed on a GPU.

[0131] For example, the analysis module can be deployed on the CPU.

[0132] For example, the analysis module can be a single module.

[0133] For example, the analysis module can be a module group that contains multiple modules.

[0134] Figure 4 This is a schematic diagram illustrating data transmission as an example.

[0135] Reference Figure 4 For example, the GPU transmits the compressed rendering information to the encoder. That is, the GPU can directly transmit the compressed rendering information to the encoder without going through any other modules. This application embodiment uses the GPU transmitting compressed rendering information to the encoder as an example for illustration.

[0136] Scene 1 One possible implementation is to compress the rendering information by reducing its resolution.

[0137] Figure 5 This is a schematic diagram illustrating a data processing procedure as an example.

[0138] S501, obtain rendering information.

[0139] For example, after the GPU performs a rendering operation and obtains the rendered image, it can transmit the rendered image to the encoder on the one hand, and obtain the rendering information on the other hand, and then compress the rendering information and transmit it to the encoder, as shown in S502~S503 below.

[0140] S502 reduces the resolution of rendered information.

[0141] For example, the resolution of the rendered information can be reduced by spatial downsampling of the rendered information.

[0142] For example, sampling ratio information can be preset, and then the GPU can perform spatial downsampling of the rendering information according to the preset sampling ratio information.

[0143] For example, sampling ratio information can be used to characterize the degree of reduction in the resolution of the rendered information.

[0144] For example, the sampling ratio information includes: horizontal sampling ratio information and vertical sampling ratio information.

[0145] For example, the GPU can also set the sampling ratio information based on the transmission efficiency of rendering information and the optimization efficiency of the encoder based on the rendering information.

[0146] For example, regarding RMV information, the encoder can utilize RMV information as the MV in the motion estimation process during encoding optimization based on RMV information. Therefore, the sampling ratio information of RMV information can be set based on MV.

[0147] In one possible approach, the sampling ratio of the RMV information can be set based on the encoder's storage granularity information for the MV. For example, the storage granularity information for the MV can refer to the resolution of the blocks for which the encoder needs to store the encoded MV during motion estimation. For instance, if the storage granularity of the MV is 4×4, it means that one MV needs to be stored for each 4×4 block in the encoded block. Similarly, if the storage granularity of the MV is 2×2, it means that one MV needs to be stored for each 2×2 block in the encoded block. For example, the storage granularity information can include horizontal pixels and vertical pixels. For example, the horizontal sampling ratio information corresponding to the RMV information can be the ratio (e.g., N1 / A1) of a number divisible by the horizontal pixels in the storage granularity information (which can be represented by N1, a positive integer, excluding the horizontal pixels in the storage granularity information) to the horizontal pixels in the storage granularity information (which can be represented by A1, a positive integer). The vertical sampling ratio information can be: the ratio of a number that can be divided by the vertical pixels in the stored granularity information (excluding the vertical pixels in the stored granularity information, it can be represented by N2, where N2 is a positive integer) to the vertical pixels in the stored granularity information (which can be represented by A2, where A2 is a positive integer) (such as N2 / A2).

[0148] For example, the storage granularity of MV information is 2×2, i.e., A1=2, A2=2. N1 can be 1, N2 can be 1, then the horizontal sampling ratio is 1 / 2, and the vertical sampling ratio is 1 / 2. As another example, the storage granularity of MV information is 4×4, i.e., A1=4, A2=4. N1 can be 1, N2 can be 1, then the horizontal sampling ratio is 1 / 4, and the vertical sampling ratio is 1 / 4. Of course, N1 can be 2, N2 can be 2, then the horizontal sampling ratio is 1 / 2, and the vertical sampling ratio is 1 / 2.

[0149] For example, after the encoder uses RMV information as MV information in the motion estimation process, the decoder needs to use the MV information for motion compensation. One possible approach is to set the sampling ratio of the RMV information based on the compensation granularity information required by the decoder when performing motion compensation on the encoded block. For example, the compensation granularity information can refer to the resolution of the block for which the decoder needs to use MV information for compensation during motion compensation. For instance, a compensation granularity of 4×4 indicates that one MV needs to be used for compensation for each 4×4 block in the encoded block, or it indicates that in the finest case during encoding, a 4×4 block can correspond to storing one MV (that is, each 4×4 block in a encoded block corresponds to storing one MV. For example, if the size of the encoded block is 4×4, then one MV is stored. Or, for example, if the size of the encoded block is 8×8, then four MVs are stored). For example, a 2×2 storage granularity for MV information indicates that one MV is needed for each 2×2 block in the coding block, or that the finest case in the coding process is that one MV can be stored for each 2×2 block (that is, one MV is stored for each 2×2 block in a coding block. For example, if the coding block size is 4×4, then 4 MVs are stored. Or, if the coding block size is 8×8, then 16 MVs are stored). For example, the compensation granularity information can include horizontal pixels and vertical pixels. An exemplary horizontal sampling ratio information corresponding to RMV information can be: the ratio of a number divisible by the horizontal pixels in the compensation granularity information (which can be represented by N3, a positive integer, excluding the horizontal pixels storing the granularity information) to the horizontal pixels in the compensation granularity information (which can be represented by A3, a positive integer) (e.g., N3 / A3). The vertical sampling ratio information can be: the ratio of the number divisible by the vertical pixels in the compensation granularity information (excluding the vertical pixels storing the granularity information, which can be represented by N4, where N4 is a positive integer) to the vertical pixels in the compensation granularity information (which can be represented by A4, where A4 is a positive integer) (such as N4 / A4).

[0150] For example, the compensation granularity information is 2×2, i.e., A3=2, A4=2. N3 can be 1, N4 can be 1, then the horizontal sampling ratio information is 1 / 2, and the vertical sampling ratio information is 1 / 2. As another example, the compensation granularity information is 4×4, i.e., A3=4, A4=4. N3 can be 1, N4 can be 1, then the horizontal sampling ratio information is 1 / 4, and the vertical sampling ratio information is 1 / 4. Of course, N3 can be 2, N4 can be 2, then the horizontal sampling ratio information is 1 / 2, and the vertical sampling ratio information is 1 / 2.

[0151] For example, this application embodiment uses N1 / A1 as the horizontal sampling ratio information and N2 / A2 as the vertical sampling ratio information as an example for illustrative purposes.

[0152] For example, the rendering information can be downsampled in both the horizontal and vertical dimensions based on the horizontal and vertical sampling ratio information to reduce the resolution of the rendering information.

[0153] One possible approach is to downsample based on sampling. For example, the rendered information can be sampled horizontally according to the horizontal sampling ratio and vertically according to the vertical sampling ratio to obtain the rendered information with reduced resolution.

[0154] For example, the resolution of RMV information is RMV information can be represented as:

[0155] Each element in the RMV information can be used... This indicates that i and j are integers, with i ranging from 0 to (H-1) and j ranging from 0 to (W-1).

[0156] For example, elements of each row of the RMV information can be sampled according to the horizontal sampling ratio information to extract... , QW = A1 / N1. For example, when N1 = 1 and A1 = 2, QW = 2 and t = 2j. When N1 = 1 and A1 = 4, QW = 4 and t = 4j. Then, according to the vertical sampling ratio information, the extracted... The elements of each column in the matrix are sampled and extracted. , QH = A² / N². For example, when N² = 1 and A² = 2, QH = 2 and s = 2i. When N² = 1 and A² = 4, QH = 4 and s = 4i. Furthermore, elements extracted from the RMV information... The RMV information after reducing the sampling rate is composed of (W / QW) × (H / QH).

[0157] For example, elements of each row of the RMV information can be sampled according to the horizontal sampling ratio information to extract... , QW = A1 / N1. For example, when N1=1 and A1=2, QW=2 and t=2j. When N1=1 and A1=4, QW=4 and t=4j. Furthermore, according to the vertical sampling ratio information, elements in each column of the RMV information are sampled and extracted. , QH = A² / N². For example, when N² = 1 and A² = 2, QH = 2 and s = 2i. When N² = 1 and A² = 4, QH = 4 and s = 4i. Then the extracted... The matrix formed and The intersection of the matrices formed yields Furthermore, elements extracted from RMV information can be used. The RMV information after reducing the sampling rate is composed of (W / QW) × (H / QH).

[0158] For example, the resolution of ID information is ID information can be represented as:

[0159] Each element in the ID information can be used Let represent , where i and j are integers, i ranges from 0 to (H-1), and j ranges from 0 to (W-1).

[0160] For example, elements in each row of the ID information can be sampled according to the horizontal sampling ratio information to extract... , QW = A1 / N1. For example, when N1 = 1 and A1 = 2, QW = 2 and t = 2j. When N1 = 1 and A1 = 4, QW = 4 and t = 4j. Then, according to the vertical sampling ratio information, the extracted... Elements in each column of the matrix are sampled and extracted. , QH = A² / N². For example, when N² = 1 and A² = 2, QH = 2 and s = 2i. When N² = 1 and A² = 4, QH = 4 and s = 4i. Furthermore, elements extracted from the ID information can be used... The ID information after reducing the sampling rate is composed of (W / QW) × (H / QH).

[0161] For example, elements in each row of the ID information can be sampled according to the horizontal sampling ratio information to extract... , QW = A1 / N1. For example, when N1 = 1 and A1 = 2, QW = 2 and t = 2j. When N1 = 1 and A1 = 4, QW = 4 and t = 4j. Furthermore, according to the vertical sampling ratio information, elements in each column of the ID information are sampled and extracted. , QH = A² / N². For example, when N² = 1 and A² = 2, QH = 2 and s = 2i. When N² = 1 and A² = 4, QH = 4 and s = 4i. Then the extracted... The matrix formed and The intersection of the matrices formed yields Furthermore, elements extracted from the ID information can be used. The ID information after reducing the sampling rate is composed of (W / QW) × (H / QH).

[0162] For example, the resolution of the depth information is Depth information can be represented as:

[0163] Each element in the Depth information can be used Let represent , where i and j are integers, i ranges from 0 to (H-1), and j ranges from 0 to (W-1).

[0164] For example, elements in each row of the Depth information can be sampled according to the horizontal sampling ratio information to extract... , QW = A1 / N1. For example, when N1 = 1 and A1 = 2, QW = 2 and t = 2j. When N1 = 1 and A1 = 4, QW = 4 and t = 4j. Then, according to the vertical sampling ratio information, the extracted... Elements in each column of the matrix are sampled and extracted. , QH = A² / N². For example, when N² = 1 and A² = 2, QH = 2 and s = 2i. When N² = 1 and A² = 4, QH = 4 and s = 4i. Furthermore, elements extracted from the Depth information... The reduced sampling rate constitutes the Depth information, where the resolution of the reduced sampling rate Depth information is (W / QW) × (H / QH).

[0165] For example, elements in each row of the Depth information can be sampled according to the horizontal sampling ratio information to extract... , QW = A1 / N1. For example, when N1=1 and A1=2, QW=2 and t=2j. When N1=1 and A1=4, QW=4 and t=4j. Furthermore, according to the vertical sampling ratio information, elements in each column of the Depth information are sampled and extracted. , QH = A² / N². For example, when N² = 1 and A² = 2, QH = 2 and s = 2i. When N² = 1 and A² = 4, QH = 4 and s = 4i. Then the extracted... The matrix formed and The intersection of the matrices formed yields Furthermore, elements extracted from the Depth information can be used to... The reduced sampling rate constitutes the Depth information, where the resolution of the reduced sampling rate Depth information is (W / QW) × (H / QH).

[0166] One possible approach is to perform downsampling based on multi-tap filtering (i.e., calculating weighted values). For example, the rendering information can be divided into multiple sub-matrices according to the horizontal and vertical sampling ratios; the elements in each sub-matrix are weighted, and the reduced-resolution rendering information is determined based on the weighted calculation results for each sub-matrix.

[0167] For example, the resolution of RMV information is RMV information can be represented as:

[0168] Each element in the RMV information can be used... This indicates that i and j are integers, with i ranging from 0 to (H-1) and j ranging from 0 to (W-1). Let x be the component. For the y component.

[0169] For example, the RMV information can be divided into QW×QH submatrices, each of which can be used... This indicates that p and q are integers, with p ranging from 0 to ((H / QH)-1) and q ranging from 0 to ((W / QW)-1). In this case, the RMV information can be represented as:

[0170] Then, for a submatrix, the weighted calculation of each element's x-component can be performed to obtain the weighted result of the x-component. Similarly, the weighted calculation of each element's y-component can be performed to obtain the weighted result of the y-component. Finally, the weighted calculation results of the x-component and the y-component can be combined to form the weighted result of the submatrix.

[0171] For example, assuming QW=2 and QH=2, then The weighted calculation result of the x-component can be:

[0172] The weighted calculation result of the y-component can be:

[0173] Where s is an integer, ranging from 0 to W / 2, and t is an integer, ranging from 0 to H / 2. p = 2s, q = 2t.

[0174] Then, the weighted calculation results of each sub-matrix are used to form the RMV information after reducing the sampling rate. The resolution of the RMV information after reducing the sampling rate is: (W / QW) × (H / QH).

[0175] For example, the resolution of ID information is ID information can be represented as:

[0176] Each element in the ID information can be used Let represent , where i and j are integers, i ranges from 0 to (H-1), and j ranges from 0 to (W-1). For R components, For the G component, For the G component.

[0177] For example, the ID information can be divided into QW×QH submatrices, each of which can be used... This indicates that p and q are integers, with p ranging from 0 to ((H / QH)-1) and q ranging from 0 to ((W / QW)-1). In this case, the ID information can be represented as:

[0178] Then, for a given submatrix, the weighted R-components of each element can be calculated to obtain the weighted result of the R-components. Similarly, the weighted G-components of each element can be calculated to obtain the weighted result of the G-components. Finally, the weighted results of the R-components, G-components, and B-components can be combined to form the weighted result of the submatrix.

[0179] For example, assuming QW=2 and QH=2, then The weighted calculation result of the R component can be:

[0180] The weighted calculation result of the G component can be:

[0181] The weighted calculation result of the B component can be:

[0182] Where s is an integer, ranging from 0 to W / 2, and t is an integer, ranging from 0 to H / 2. p = 2s, q = 2t.

[0183] Then, the weighted calculation results of each sub-matrix are used to form the ID information after reducing the sampling rate. The resolution of the ID information after reducing the sampling rate is: (W / QW)×(H / QH).

[0184] For example, the resolution of the depth information is Depth information can be represented as:

[0185] Each element in the Depth information can be used Let represent , where i and j are integers, i ranges from 0 to (H-1), and j ranges from 0 to (W-1).

[0186] For example, the depth information can be divided into QW×QH submatrices. Each submatrix can be used... This indicates that p and q are integers, with p ranging from 0 to ((H / QH)-1) and q ranging from 0 to ((W / QW)-1). In this case, the Depth information can be represented as:

[0187] Then, for a submatrix, each element in the submatrix can be weighted to obtain the weighted result of the submatrix.

[0188] For example, suppose QW=2, QH=A2=2, then The weighted calculation result can be:

[0189] Where s is an integer, ranging from 0 to W / 2, and t is an integer, ranging from 0 to H / 2. p = 2s, q = 2t.

[0190] Then, the weighted calculation results of each sub-matrix are used to form the Depth information after reducing the sampling rate. The resolution of the Depth information after reducing the sampling rate is: (W / QW) × (H / QH).

[0191] S503 transmits rendering information after the resolution has been reduced.

[0192] After the GPU reduces the resolution of the rendering information, the reduced-resolution rendering information can be transmitted to the encoder.

[0193] Figure 6 This is a schematic diagram illustrating a data transmission process as an example.

[0194] Figure 6 In the process, the resolution of the rendering information 601 obtained by the GPU is 16×16. Assuming the horizontal sampling ratio is 1 / 2 and the vertical sampling ratio is 1 / 2, after reducing the resolution of the rendering information, we can obtain the reduced-resolution rendering information 602, which has a resolution of 8×8. Then, the reduced-resolution rendering information 602 is sent to the encoder.

[0195] In summary, by reducing the resolution, the amount of rendering information transmitted from the GPU to the encoder is reduced, thereby lowering the transmission latency of the rendering information. Furthermore, since the encoder encodes the rendering information only after receiving it, this embodiment of the application can reduce the encoder's waiting time for encoding, thus reducing the overall transmission latency of the cloud rendering process. Additionally, reducing the amount of rendering information also reduces the bandwidth requirements for its transmission. Moreover, the cache space used to store rendering information in the encoder can be reduced, thereby saving encoder cache space.

[0196] Scene 2 For example, the precision of the rendering information required by the encoder for encoding optimization is lower than the precision of the rendering information obtained by the GPU. Therefore, one possible implementation is to compress the rendering information by reducing its precision.

[0197] Figure 7 This is a schematic diagram illustrating data processing as an example.

[0198] S701, obtain rendering information.

[0199] S701 is similar to S501 mentioned above, and will not be described again here.

[0200] S702 reduces the bit depth of rendering information.

[0201] For example, the precision of rendering information is essentially the bit depth of the rendering information; thus, the precision of the rendering information can be reduced by decreasing the bit depth of the rendering information, thereby compressing the rendering information.

[0202] For example, assuming the bit depth of the rendering information is a first bit depth value, a second bit depth value can be obtained, wherein the second bit depth value is less than the first bit depth value. Then, the bit depth of each element in the rendering information is converted from the first bit depth value to the second bit depth value.

[0203] For example, the second bit depth value can be determined based on the bit depth of the rendering information required for encoder encoding optimization (e.g., the corresponding value is the third bit depth value) and the first bit depth value. Optionally, the second bit depth value can be equal to the third bit depth value. Optionally, the second bit depth value can be greater than the third bit depth value and less than the first bit depth value.

[0204] For example, the bit depth of each element in the rendering information can be converted from a first bit depth value to a second bit depth value by retaining the high-order bits.

[0205] For example, the bit depth of each element in the rendering information can be converted from a first bit depth value to a second bit depth value by truncating the high bits.

[0206] For example, the first bit depth value of different rendering information can be the same or different. The second bit depth value corresponding to different rendering information can be the same or different; it can be set according to requirements, and this application embodiment does not limit this.

[0207] For example, for RMV information, if the first bit depth is 32 bits and the second bit depth is 16 bits, the bit depth of the RMV information can be converted from 32 bits to 16 bits. Specifically, the bit depth of each element contained in the RMV information can be converted from 32 bits to 16 bits. Specifically, the bit depth of the x-component of each element in the RMV information can be converted from 32 bits to 16 bits, and the bit depth of the y-component of each element in the RMV information can also be converted from 32 bits to 16 bits. Specific conversion methods can control the bit depth by removing lower-order bits, such as reducing the number of significant digits after the decimal point.

[0208] For example, for ID information, if the first bit depth is 8 bits and the second bit depth is 4 bits, then the bit depth of the ID information can be converted from 8 bits to 4 bits. Specifically, the bit depth of each element in the ID information can be converted from 8 bits to 4 bits. Specifically, the bit depth of the R component, G component, and B component of each element in the ID information can be converted from 8 bits to 4 bits. The conversion method can be to remove the lower-order bits or use quantization.

[0209] For example, for Depth information, if the first bit depth is 8 bits and the second bit depth is 4 bits, then the bit depth of the Depth information can be converted from 8 bits to 4 bits. Specifically, the bit depth of each element contained in the Depth information can be converted from 8 bits to 4 bits.

[0210] S703 transmits rendering information after reducing the bit depth.

[0211] After the GPU reduces the bit depth of the rendering information, the reduced bit depth rendering information can be transmitted to the encoder.

[0212] Figure 8 This is a schematic diagram illustrating a data transmission process as an example.

[0213] Figure 8 In the rendering information 801 obtained by the GPU, the first bit depth value is 32 bits. Assuming the second bit depth value is 16 bits, reducing the bit depth of the rendering information yields the reduced bit depth rendering information 802, which has a bit depth of 16 bits. The reduced bit depth rendering information 802 is then sent to the encoder.

[0214] In summary, by reducing the bit depth, the amount of rendering information transmitted from the GPU to the encoder is reduced, thereby lowering the transmission latency of the rendering information. Furthermore, since the encoder encodes the rendering information only after receiving it, this embodiment can reduce the encoder's waiting time for encoding, thus reducing the overall transmission latency of the cloud rendering process. Additionally, reducing the amount of rendering information also reduces the bandwidth requirements for its transmission. Moreover, the cache space used to store rendering information in the encoder can be reduced, further conserving encoder cache space.

[0215] Scene 3 As described above, the encoder encodes the rendered image in blocks. Therefore, in one possible implementation, the rendering information is transmitted in blocks. When the encoder receives any block of rendering information, it can encode the macroblock corresponding to that block. This allows the transmission of rendering information and encoding to occur synchronously, thereby reducing the time the encoder spends waiting for rendering information. Furthermore, the data size of a block is smaller than that of the complete rendering information, thus the embodiments of this application can also reduce the bandwidth requirements for the transmission of rendering information.

[0216] For example, in this embodiment of the application, the GPU can transmit rendering information in blocks to the analysis module, or it can transmit rendering information in blocks to the encoder; see above. Figure 2 The description of the above will not be repeated here. Embodiment 1 of this application illustrates how the GPU transmits rendering information in blocks to the encoder.

[0217] Figure 9 This is a schematic diagram illustrating a data processing procedure as an example.

[0218] S901, obtain rendering information.

[0219] S901 is similar to S501 mentioned above, so it will not be described again here.

[0220] S902 divides the rendering information into multiple information blocks.

[0221] For example, the encoder can pre-set a block division method to divide the rendered image into multiple macroblocks. Subsequently, before encoding, the encoder can divide the rendered image into blocks according to this block division method.

[0222] For example, in the same application scenario, the encoder can use the same method to segment all rendered images. In different application scenarios, the encoder can use the same or different methods to segment the rendered images, depending on the requirements. This application embodiment does not impose any restrictions on this.

[0223] For example, after the encoder sets the chunking method to divide the rendered image into multiple macroblocks, the chunking method can be sent to the GPU.

[0224] For example, the GPU can actively obtain the tile format set by the encoder from the encoder. Optionally, the GPU can obtain the tile format set by the encoder from the encoder before performing the rendering operation. Optionally, the GPU can obtain the tile format set by the encoder from the encoder after obtaining the rendering information.

[0225] For example, the GPU pre-defines the encoder's segmentation method for the rendered image, so the GPU can directly use the pre-defined segmentation method.

[0226] For example, after obtaining the encoder's segmentation method for the rendered image, the GPU can segment the rendering information according to the segmentation method to obtain multiple information blocks.

[0227] Figure 10 This is a schematic diagram of a block structure as an example.

[0228] Reference Figure 10 (1), Figure 10 (1) is the rendered image, which is divided into 9 macroblocks: C1, C2, C3, C4, C5, C6, C7, C8 and C9.

[0229] Reference Figure 10 (2), Figure 10 (2) Rendering information. The GPU, according to... Figure 10 (1) The rendering image is divided into 9 information blocks: R1, R2, R3, R4, R5, R6, R7, R8 and R9.

[0230] S903 transmits each information block separately.

[0231] For example, after the encoder divides the rendered image into multiple macroblocks, it can set the encoding order for multiple macroblocks. Subsequently, each macroblock can be encoded according to the encoding order of the macroblocks.

[0232] For example, in the same application scenario, the encoder may encode multiple macroblocks in the same order. In different application scenarios, the encoder may encode multiple macroblocks in the same or different order, depending on the requirements. This application embodiment does not impose any restrictions on this.

[0233] For example, after the encoder sets the encoding order for multiple macroblocks, it can send the encoding order to the GPU.

[0234] For example, the GPU can actively obtain the encoding order of multiple macroblocks from the encoder. Optionally, the GPU can obtain the encoding order of multiple macroblocks from the encoder before performing rendering operations. Optionally, the GPU can obtain the encoding order of multiple macroblocks from the encoder after obtaining rendering information.

[0235] For example, the encoding order of multiple macroblocks is pre-defined in the GPU, so the GPU can directly use the pre-defined encoding order.

[0236] For example, the GPU can determine the transmission order of the information blocks according to a preset encoding order. (See again...) Figure 10 (1) If Figure 10(1) The encoding order of the 9 macroblocks in the rendered image is: C1→C2→C3→C4→C5→C6→C7→C8→C9, then Figure 10 (2) The transmission order of the 9 information blocks for rendering information can be: R1→R2→R3→R4→R5→R6→R7→R8→R9.

[0237] For example, the GPU can send each information block sequentially according to the transmission order, thereby serially transmitting each information block to the encoder. Correspondingly, the encoder can receive each information block sequentially.

[0238] Figure 11 This is a schematic diagram illustrating an exemplary information block transmission process. (Refer to...) Figure 11 The GPU divides the rendering information 1101 into nine information blocks 1102 (including: R1, R2, R3, R4, R5, R6, R7, R8, and R9), and then transmits R1, R2, R3, R4, R5, R6, R7, R8, and R9 to the encoder in the order R1→R2→R3→R4→R5→R6→R7→R8→R9. Correspondingly, the encoder can receive R1 first, then R2, then R3, and so on.

[0239] For example, the GPU can send multiple information blocks simultaneously (multiple consecutively numbered information blocks, such as C1 and C2, C3 and C4, and so on) according to the transmission order of the information blocks, thereby transmitting these multiple information blocks to the encoder in parallel. Since the transmission bandwidth between the GPU and the encoder is limited, the encoder waits less time to receive rendering information compared to parallel transmission of information blocks. This application embodiment uses the serial transmission of information blocks as an example for illustrative purposes.

[0240] For example, after the GPU sends the current information block, it can wait for a set duration before sending the next information block. The set duration is determined based on the transmission time of an information block and the encoding time of the encoder for a macroblock. It is only necessary to ensure that the next information block is transmitted to the encoder before or simultaneously with the encoder's encoding of the next macroblock.

[0241] For example, the duration can be set to 0, in which case the GPU can send the next information block immediately after sending one information block.

[0242] For example, the set duration can be the transmission duration of one information block. In this case, the GPU sends out one information block and confirms that the information block has been successfully received by the encoder before sending the next information block. When the encoder detects that an information block has been received, it can return a response to the GPU to inform the GPU that it has successfully received the information block.

[0243] For example, the set duration can be equal to the difference between the encoding duration of a macroblock and the transmission duration of an information block. In this case, the GPU can wait for the difference duration after sending out an information block before sending the next information block, so that the next information block is transmitted to the encoder at the same time as the encoder is encoding the next macroblock.

[0244] For example, the set duration can be less than the difference between the encoding duration of a macroblock and the transmission duration of an information block. In this case, the GPU can wait for the difference duration after sending an information block before sending the next information block, so that the next information block can be transmitted to the encoder before the encoder encodes the next macroblock.

[0245] The following example illustrates the transmission duration of a single information block.

[0246] Figure 12 This is an illustrative diagram of transmission and encoding. Figure 12 In this process, the encoder performs serial encoding.

[0247] For ease of explanation, assume that the transmission duration of an information block is Rt, the encoding duration of a macroblock is Ct, there are 9 information blocks, and 9 macroblocks (the block division method for information blocks and macroblocks can be found in [reference]). Figure 10 The transmission order of information blocks and the encoding order of macroblocks can be referred to Figure 11 ).

[0248] Figure 12 For example, Rt:Ct=5:8. Here, the time difference between Rt0 and Rt1, the time difference between Rt1 and Rt2, the time difference between Rt2 and Rt3, ... are all Rt, and the time difference between Rt1 and Ct1, the time difference between Ct1 and Ct2, the time difference between Ct2 and Ct3, ... are all Ct.

[0249] Figure 12 In this process, the GPU begins transmitting rendering information at time Rt0, specifically: Rt0 time: GPU sends information block R1.

[0250] Time Rt1: The encoder receives information block R1 and begins encoding macroblock C1; The GPU begins sending information block R2.

[0251] Rt2 time: The encoder received information block R2; The GPU begins sending information block R3.

[0252] Time Ct1: The encoder has finished encoding C1 and begins encoding C2.

[0253] Rt3 time: The encoder received information block R3; The GPU begins sending information block R4.

[0254] Rt4 time: The encoder received information block R4; The GPU begins sending information block R5.

[0255] Ct2 time: The encoder has finished encoding C2 and begins encoding C3.

[0256] Rt5 time: The encoder received information block R5; The GPU begins sending information block R6.

[0257] Ct3 time: The encoder has finished encoding C3 and is now starting to encode C4.

[0258] Rt6 time: The encoder received information block R6; The GPU begins sending information block R7.

[0259] Rt7 time: The encoder received information block R7; The GPU begins sending information block R8.

[0260] Ct4 time: The encoder has finished encoding C4 and is now starting to encode C5.

[0261] Rt8 time: The encoder received information block R8; The GPU begins sending information block R9.

[0262] Ct5 time: The encoder has finished encoding C5 and is now starting to encode C6.

[0263] Rt9 time: The encoder received information block R9.

[0264] Ct6 time: The encoder has finished encoding C6 and is now starting to encode C7.

[0265] Ct7 time: The encoder has completed encoding C7 and begins encoding C8.

[0266] Ct8 time: The encoder has completed encoding C8 and begins encoding C9.

[0267] Ct9 time: The encoder completes the encoding of C9.

[0268] For example, after the encoder completes the encoding optimization based on an information block, it can release the cache of that information block to save the encoder's cache space.

[0269] For example, at time Ct1, after the encoder completes encoding C1 and optimizes the encoding based on R1, the buffer space in the encoder used to store R1 can be released. At time Ct2, after the encoder completes encoding C2 and optimizes the encoding based on R2, the buffer space in the encoder used to store R2 can be released, and so on.

[0270] It should be noted that if the GPU has already generated the rendered image for the next frame at time Rt9, then the rendered image for the next frame can be sent at time Rt9, and the rendering information of the rendered image for the next frame can be divided into blocks. Then, after the rendered image for the next frame is transmitted to the encoder, it is referenced... Figure 12 The corresponding description and the information block that sends the rendering information of the next frame's rendered image will not be elaborated here.

[0271] It should be noted that if, before the encoder completes the encoding of C9 at time Ct9, it receives the rendered image of the next frame and at least one information block corresponding to the rendering information of the rendered image of the next frame, then the encoder can, at time Ct9, proceed according to the reference... Figure 12 The corresponding description and the encoding of macroblocks of the next frame's rendered image will not be elaborated here. In other words, if the encoder receives at least one information block of the next frame's rendered image and its corresponding rendering information during the encoding of the current frame's rendered image, then after completing the encoding of the current frame's rendered image, the rendering information can directly encode the next frame's rendered image without waiting for Rt duration.

[0272] Figure 13 This is an illustrative diagram of transmission and encoding. Figure 13 In this context, the encoder performs parallel encoding, such as 2-way parallel encoding.

[0273] For ease of explanation, assume that the transmission duration of an information block is Rt, the encoding duration of a macroblock is Ct, there are 9 information blocks, and 9 macroblocks (the block division method for information blocks and macroblocks can be found in [reference]). Figure 10 The transmission order of information blocks and the encoding order of macroblocks can be referred to Figure 11 ).

[0274] Figure 13 In the example, Rt:Ct=5:8. Here, the time difference between Rt0 and Rt1, the time difference between Rt1 and Rt2, the time difference between Rt2 and Rt3, ..., is all Rt, and the time difference between Rt9 and Ct1 is Ct.

[0275] Figure 12 In this process, the GPU begins transmitting rendering information at time Rt0, specifically: Rt0 time: GPU sends information block R1.

[0276] Time Rt1: The encoder receives information block R1 and begins encoding macroblock C1; The GPU begins sending information block R2.

[0277] Rt2 time: The encoder receives information block R2 and begins encoding macroblock C2; The GPU begins sending information block R3.

[0278] Rt3 time: The encoder receives information block R3 and begins encoding macroblock C3; The GPU begins sending information block R4.

[0279] Rt4 time: The encoder receives information block R4 and begins encoding macroblock C4; The GPU begins sending information block R5.

[0280] Rt5 time: The encoder receives information block R5 and begins encoding macroblock C5; The GPU begins sending information block R6.

[0281] Rt6 time: The encoder receives information block R6 and begins encoding macroblock C6; The GPU begins sending information block R7.

[0282] Rt7 time: The encoder receives information block R7 and begins encoding macroblock C7; The GPU begins sending information block R8.

[0283] Rt8 time: The encoder receives information block R8 and begins encoding macroblock C8; The GPU begins sending information block R8.

[0284] Rt9 time: The encoder receives information block R9 and begins encoding macroblock C9; Time Ct1: The encoder completes the encoding of C9.

[0285] It should be noted that if the GPU has already generated the rendered image for the next frame at time Rt9, then the rendered image for the next frame can be sent at time Rt9, and the rendering information of the rendered image for the next frame can be divided into blocks and numbered. Then, after the rendered image for the next frame is transmitted to the encoder, it is referenced... Figure 10 The corresponding description and the information block that sends the rendering information of the next frame's rendered image will not be elaborated here.

[0286] It should be noted that if, before the encoder completes the encoding of C9 at time Ct9, it receives the rendered image of the next frame and at least one information block corresponding to the rendering information of the rendered image of the next frame, then the encoder can, at time Ct9, proceed according to the reference... Figure 10 The corresponding description and the encoding of information blocks for the next frame's rendered image will not be elaborated here. In other words, if the encoder receives at least one information block of the next frame's rendered image and its corresponding rendering information during the encoding of the current frame's rendered image, then after completing the encoding of the current frame's rendered image, the rendering information can directly encode the next frame's rendered image without waiting for Rt duration.

[0287] based on Figure 12 and Figure 13 As can be seen, in this embodiment, if the encoder does not receive the information block corresponding to the rendering information of the (n+1)th frame during the encoding of the image rendered from the nth frame (n is a positive integer), the encoder, after completing the encoding of the image rendered from the nth frame, waits for T=Rt (i.e., the transmission time of one information block) before starting to encode the image rendered from the (n+1)th frame. However, existing technologies require waiting... (That is, the total transmission time of the rendered information). In other words, compared with the prior art, in this case, the encoder waiting time in the embodiments of this application is reduced. .

[0288] In this embodiment, if the encoder receives at least one information block containing the rendered image of the (n+1)th frame and corresponding rendering information during the encoding process of the image rendered in the nth frame, the encoder can immediately begin encoding the image rendered in the (n+1)th frame after completing the encoding of the image rendered in the nth frame. In contrast, existing technologies require waiting... In other words, compared to existing technologies, the encoder waiting time in this embodiment of the application is reduced. .

[0289] In the embodiments described above, the GPU transmits rendering information in blocks according to the encoding order of macroblocks in the rendered image by the encoder. This allows the transmission of encoding and rendering information to proceed synchronously, reducing the encoder's waiting time for encoding and thus reducing the transmission latency and bandwidth requirements of the rendering information. Furthermore, the encoder performs encoding only after receiving the rendering information; therefore, this embodiment reduces the encoder's waiting time for encoding, thereby reducing the overall transmission latency of the cloud rendering process. In addition, after the encoder completes encoding optimization based on an information block, it releases the cache of that information block, saving encoder cache space.

[0290] Scene 4 For example, the two methods of compressing rendering information mentioned above can be combined to compress the rendering information, thereby further reducing the amount of rendering information data, which in turn reduces the transmission latency and bandwidth requirements of the rendering information, and further saves the cache space used by the encoder to store the rendering information.

[0291] One possible implementation is to first reduce the resolution of the rendered information, and then reduce the bit depth of the rendered information.

[0292] Figure 14 This is a schematic diagram illustrating a data processing procedure as an example.

[0293] S1401, Obtain first rendering information.

[0294] For ease of explanation, the rendering information obtained by the GPU can be referred to as the first rendering information.

[0295] S1402 is similar to S501 above, and will not be described again here.

[0296] S1402, reduce the resolution of the first rendering information to obtain the second rendering information.

[0297] Step S1402 is similar to S502 above, and will not be repeated here.

[0298] Among these, the rendering information after reducing the resolution can be referred to as the second rendering information.

[0299] S1403, reduce the bit depth of the second rendering information to obtain the third rendering information.

[0300] Step S1403 is similar to S702 above, and will not be repeated here.

[0301] Among them, the rendering information after reducing the resolution and bit depth can be referred to as the third rendering information.

[0302] S1404, transmits third-party rendering information.

[0303] Figure 15 This is a schematic diagram illustrating a data transmission process as an example.

[0304] Reference Figure 15 Assume the GPU-acquired rendering information (i.e., the first rendering information) 1501 has a resolution of 4×4 and a bit depth of 32 bits. If the horizontal sampling ratio is 1 / 2 and the vertical sampling ratio is 1 / 2, then reducing the resolution of the rendering information yields the reduced-resolution rendering information (i.e., the second rendering information) 1502, with a resolution of 2×2 and a bit depth of 32 bits. If the second bit depth is 16 bits, then reducing the bit depth of the reduced-resolution rendering information 1502 yields the reduced-resolution and reduced-bit-depth rendering information (i.e., the third rendering information) 1503, with a resolution of 2×2 and a bit depth of 16 bits. The third rendering information 1503 is then transmitted to the encoder.

[0305] Scene 5 One possible implementation is to first reduce the bit depth of the rendering information, and then reduce the resolution of the rendering information.

[0306] Figure 16 This is a schematic diagram illustrating a data processing procedure as an example.

[0307] S1601, Obtain the first rendering information.

[0308] For ease of explanation, the rendering information obtained by the GPU can be referred to as the first rendering information.

[0309] S1602 is similar to S501 above, and will not be described again here.

[0310] S1602. Reduce the bit depth of the first rendering information to obtain the fourth rendering information.

[0311] Step S1602 is similar to S702 above, and will not be repeated here.

[0312] Among them, the rendering information after reducing the bit depth can be called the fourth rendering information.

[0313] S1603. Reduce the bit depth of the fourth rendering information to obtain the third rendering information.

[0314] Step S1603 is similar to S502 above, and will not be repeated here.

[0315] Among them, the rendering information after reducing the resolution and bit depth can be referred to as the third rendering information.

[0316] S1604, Transmit third-party rendering information.

[0317] Figure 17 This is a schematic diagram illustrating a data transmission process as an example.

[0318] Reference Figure 17 Assume the GPU-acquired rendering information (i.e., the first rendering information) 1701 has a resolution of 4×4 and a bit depth of 32 bits. If the second bit depth is 16 bits, then reducing the bit depth of the first rendering information 1701 yields the reduced-bit-depth rendering information (i.e., the fourth rendering information) 1702, with a resolution of 4×4 and a bit depth of 16 bits. If the horizontal sampling ratio is 1 / 2 and the vertical sampling ratio is 1 / 2, then reducing the bit depth of the reduced-bit-depth rendering information yields the reduced-resolution and reduced-bit-depth rendering information (i.e., the third rendering information) 1703, with a resolution of 2×2 and a bit depth of 16 bits. The third rendering information 1703 is then transmitted to the encoder.

[0319] Scene Six One possible approach is to first divide the rendering information into multiple information blocks, then compress the information blocks and transmit them in chunks. This can reduce the amount of data transmitted for each information block, further reduce the transmission latency and bandwidth requirements of the rendering information, and save the encoder's cache space for storing information blocks.

[0320] Figure 18 This is a schematic diagram illustrating a data processing procedure as an example.

[0321] S1801, Obtain rendering information.

[0322] S1802 is similar to S501 above, and will not be described again here.

[0323] S1802, Divide the rendering information into multiple information blocks.

[0324] Step S1802 is similar to S902 above, and will not be repeated here.

[0325] S1803. Compress each information block separately.

[0326] One possible approach is to reduce the resolution of each information block only.

[0327] For example, the resolution of each information block can be reduced by referring to the description of reducing the resolution of rendering information in Scenario 1 above. For example, refer to... Figure 10The rendering information is divided into 9 information blocks. If the resolution of the rendering information is W×H, then the resolution of each information block is L×M, where W=3L and H=3M. If the horizontal ratio is N1 / A1 and the vertical ratio is N2 / A2, then after reducing the resolution of each information block, the resolution of the reduced information block is... .

[0328] For example, if the resolution of information block R1 is 32×32, and the horizontal sampling ratio is 1 / 2 and the vertical sampling ratio is 1 / 2, then information block R1 is downsampled according to the horizontal and vertical sampling ratios of 1 / 2, resulting in a reduced-resolution information block R1 with a resolution of 16×16.

[0329] One possible approach is to reduce the bit depth of each information block.

[0330] One possible approach is to first reduce the resolution of each information block, and then reduce the bit depth of each information block.

[0331] One possible approach is to first reduce the bit depth of each information block, and then reduce the resolution of each information block.

[0332] S1804. Transmit each compressed information block separately.

[0333] S1804 is similar to S903 in Scenario 3, so it will not be described in detail here.

[0334] Figure 19 This is a schematic diagram illustrating a data transmission process as an example.

[0335] Reference Figure 19 Assume the GPU acquires rendering information 1901 with a resolution of 16×16 and a bit depth of 32 bits (the first bit depth value). This rendering information can be divided into multiple information blocks 1902 (including four blocks), each with a resolution of 8×8 and a bit depth of 32 bits. Assuming a horizontal sampling ratio of 1 / 2 and a vertical sampling ratio of 1 / 2, the resolution of each information block can be reduced to obtain information blocks 1903 with a resolution of 4×4 and a bit depth of 32 bits. Assuming a second bit depth value of 16 bits, the bit depth of each information block 1903 can be reduced to obtain information blocks 1904 with reduced resolution and bit depth, each with a resolution of 4×4 and a bit depth of 16 bits. These four information blocks 1904 are then transmitted to the encoder, which can then sequentially receive information blocks R1, R2, R3, and R4 with a resolution of 4×4 and a bit depth of 16 bits.

[0336] Figure 20 This is a schematic diagram illustrating a data transmission process as an example.

[0337] Reference Figure 20 Assume the GPU acquires rendering information 2001 with a resolution of 16×16 and a bit depth of 32 bits (first bit depth value). The rendering information can be divided into multiple information blocks 1902 (including four blocks), each with a resolution of 8×8 and a bit depth of 32 bits. Assuming a second bit depth of 16 bits, the bit depth of each information block 1902 can be reduced to obtain information blocks 1903 with a reduced bit depth, each with a resolution of 8×8 and a bit depth of 16 bits. Assuming the horizontal sampling ratio is 1 / 2 and the vertical sampling ratio is 1 / 2, the resolution of each information block with reduced bit depth can be further reduced to obtain information blocks 1904 with reduced resolution and bit depth, each with a resolution of 4×4 and a bit depth of 16 bits. These four information blocks 1904 are then transmitted to the encoder, which can then sequentially receive information blocks R1, R2, R3, and R4 with a resolution of 4×4 and a bit depth of 16 bits.

[0338] Scene 7 One possible approach is to first compress the rendering information and then transmit the compressed rendering information in chunks. This can reduce the amount of data transmitted for each piece of information, further reduce the transmission latency and bandwidth requirements of the rendering information, and save the encoder's cache space for information chunks.

[0339] Figure 21 This is a schematic diagram illustrating a data processing procedure as an example.

[0340] S2101, Obtain rendering information.

[0341] S2101 is similar to S501 above, and will not be described again here.

[0342] S2102, Compressed rendering information.

[0343] One possible approach is to first reduce the resolution of each information block, and then reduce the bit depth of each information block. Refer to the description in Scenario 4 above; it will not be repeated here.

[0344] One possible approach is to first reduce the bit depth of each information block, and then reduce the resolution of each information block. Refer to the description in Scenario 5 above; it will not be repeated here.

[0345] One possible approach is to reduce the resolution of each information block, which is similar to the scenario described above and will not be repeated here.

[0346] One possible approach is to reduce the bit depth of each information block, which is similar to the scenario described in Scenario 2 above, and will not be repeated here.

[0347] S2103. Divide the compressed rendering information into multiple information blocks.

[0348] Step S2103 is similar to S902 above, and will not be repeated here.

[0349] For example, the resolution of the rendered information after resolution reduction is smaller than the resolution of the corresponding rendered image. Let's assume the resolution of the rendered image is W×H, and the resolution of the rendered information after resolution reduction is (W / QW)×(H / QH). The rendered image is divided into multiple macroblocks with a resolution of L×M, where L and M are integers greater than 1. The resolution ratio of the macroblock to the rendered image can be calculated as (L / W)×(M / H). Based on this ratio and the resolution of the rendered information after resolution reduction, the resolution of each information block can be calculated. Then, based on the resolution of each information block, the rendered information after reducing the resolution is divided into multiple information blocks.

[0350] For example, assuming W=64 and H=64, the resolution of the rendered image is 64×64. Assuming QH=2 and QW=2, the resolution of the rendered information after reducing the resolution is 32×32. Assuming L=16 and M=16, the resolution of the macroblock is 16×16, which means that the rendered image is divided into 4 macroblocks of 16×16. The resolution ratio of the macroblock to the rendered image can be calculated as: (1 / 4) × (1 / 4). Based on the resolution ratio of the macroblock to the rendered image (1 / 4) × (1 / 4), and the resolution of the rendered information after resolution reduction being 32 × 32, the resolution of the information block can be calculated to be 8 × 8. Therefore, the rendered information after resolution reduction can be divided into four 8 × 8 information blocks, one 8 × 8 information block, and one 16 × 16 macroblock.

[0351] S2104. Transmit each information block separately.

[0352] S2104 is similar to S903 in Scenario 3, so it will not be described in detail here.

[0353] Figure 22This is a schematic diagram illustrating a data transmission process as an example.

[0354] Reference Figure 22 Assume the GPU acquires rendering information 2201 with a resolution of 16×16 and a bit depth of 32 bits. If the horizontal sampling ratio is 1 / 2 and the vertical sampling ratio is 1 / 2, then reducing the resolution of the rendering information yields reduced-resolution rendering information 2202 with a resolution of 8×8 and a bit depth of 32 bits. If the second bit depth is 16 bits, reducing the bit depth of the reduced-resolution rendering information 2202 yields reduced-resolution and reduced-bit-depth rendering information 2203 with a resolution of 8×8 and a bit depth of 16 bits. This reduced-resolution and reduced-bit-depth rendering information can then be divided into multiple information blocks 2204 (including four information blocks), each with a resolution of 4×4 and a bit depth of 16 bits. These four information blocks 2204 are then transmitted to the encoder, which can then sequentially receive information blocks R1, R2, R3, and R4 with a resolution of 4×4 and a bit depth of 16 bits.

[0355] Figure 23 This is a schematic diagram illustrating a data transmission process as an example.

[0356] Reference Figure 23 Assume the GPU acquires rendering information 2301 with a resolution of 16×16 and a bit depth of 32 bits (first bit depth value). If the second bit depth value is 16 bits, the bit depth of rendering information 2301 is reduced to obtain rendering information 2202 with a reduced bit depth, having a resolution of 16×16 and a bit depth of 16 bits. If the horizontal sampling ratio is 1 / 2 and the vertical sampling ratio is 1 / 2, reducing the bit depth and resolution of the rendering information yields rendering information 2203 with a reduced resolution and bit depth, having a resolution of 8×8 and a bit depth of 16 bits. This reduced-resolution and reduced-bit-depth rendering information can then be divided into multiple information blocks 2204 (including four information blocks), each with a resolution of 4×4 and a bit depth of 16 bits. These four information blocks 2204 are then transmitted to the encoder, which can then sequentially receive information blocks R1, R2, R3, and R4 with a resolution of 4×4 and a bit depth of 16 bits.

[0357] The following describes an apparatus provided by an embodiment of this application. For example... Figure 24 As shown: In one example, Figure 24A schematic block diagram illustrating an embodiment of the present application shows an apparatus 2400. The apparatus 2400 may include a processor 2401 and a transceiver / transceiver pin 2402, and optionally, a memory 2403.

[0358] The various components of device 2400 are coupled together via bus 2404, which includes a data bus, a power bus, a control bus, and a status signal bus. However, for clarity, all buses are referred to as bus 2404 in the figure.

[0359] Optionally, the memory 2403 can be used for the instructions in the foregoing method embodiments. The processor 2401 can be used to execute the instructions in the memory 2403, control the receive pin to receive signals, and control the transmit pin to transmit signals.

[0360] Device 2400 may be an electronic device or a chip of an electronic device in the above method embodiments.

[0361] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0362] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the data processing method described in the above embodiment.

[0363] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the data processing method described in the above embodiment.

[0364] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the data processing methods in the above-described method embodiments.

[0365] In this embodiment, the electronic device, computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0366] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0367] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0368] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0369] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0370] Any content in the various embodiments of this application, as well as any content in the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0371] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0372] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0373] The steps of the methods or algorithms described in conjunction with the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0374] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0375] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: The graphics processor acquires rendering information related to the rendered image, including information used to assist the execution of the rendering process, information generated during the rendering process, or information input to the graphics processor. The graphics processor compresses the rendering information based on the parameters required during the encoder's encoding optimization process and transmits the compressed rendering information so that the encoder can perform encoding optimization based on the compressed rendering information during the encoding process of the rendered image.

2. The method according to claim 1, characterized in that, The compression of the rendering information includes: Reduce the resolution of the rendered information; and / or, Reduce the bit depth of the rendering information.

3. The method according to claim 2, characterized in that, Reducing the resolution of the rendered information includes: Obtain sampling ratio information, which includes horizontal sampling ratio information and vertical sampling ratio information; Based on the horizontal and vertical sampling ratio information, the rendering information is downsampled in both the horizontal and vertical dimensions.

4. The method according to claim 3, characterized in that, The acquisition of sampling ratio information includes: Obtain the encoder's storage granularity information for motion vectors, and determine the sampling ratio information based on the storage granularity information; or, Obtain the compensation granularity information required for motion compensation of the decoder, and determine the sampling ratio information based on the compensation granularity information.

5. The method according to claim 2, characterized in that, The bit depth of the rendering information is a first bit depth value; reducing the bit depth of the rendering information includes: Obtain a second bit depth value, which is lower than the first bit depth value; The bit depth of the rendering information is converted from a first bit depth value to a second bit depth value.

6. The method according to claim 5, characterized in that, The process of obtaining the second bit depth value includes: Obtain a third bit depth value, which is used to characterize the bit depth of the rendering information required for encoder encoding optimization; The second bit depth value is determined based on the first bit depth value and the third bit depth value.

7. The method according to claim 1, characterized in that, The transmission of the compressed rendering information includes: The compressed rendering information is divided into multiple information blocks; Each of the multiple information blocks is transmitted separately.

8. The method according to claim 1, characterized in that, The method further includes: The rendering information is divided into multiple information blocks; The compression of the rendering information includes: Each information block in the plurality of information blocks is compressed separately; The transmission of the compressed rendering information includes: Each compressed information block is transmitted separately.

9. The method according to claim 7, characterized in that, The process of dividing the compressed rendering information into multiple information blocks includes: The compressed rendering information is divided into blocks according to a preset block division method to obtain the multiple information blocks; The block segmentation method refers to the way the encoder divides the rendered image into multiple macroblocks.

10. The method according to claim 9, characterized in that, The separate transmission of each information block includes: The transmission order of each information block is determined according to a preset encoding order, wherein the encoding order is the encoding order of the encoder for the multiple macroblocks; The information blocks are transmitted sequentially according to their respective transmission order.

11. The method according to claim 1, characterized in that, The transmission of the compressed rendering information includes: Transmit the compressed rendering information to the encoder; or, The compressed rendering information is transmitted to the analysis module, which analyzes the compressed rendering information to determine encoding optimization information and transmits the encoding optimization information to the encoder. The encoder then performs encoding optimization based on the encoding optimization information during the encoding process of the rendered image.

12. A data processing method, characterized in that, include: The graphics processor acquires rendering information related to the rendered image, including information used to assist the execution of the rendering process, information generated during the rendering process, or information input to the graphics processor. The graphics processor divides the rendering information into multiple information blocks and transmits each information block separately, so that the encoder can optimize the encoding of the macroblocks corresponding to each information block based on each information block during the encoding process of the rendered image. The step of dividing the rendering information into multiple information blocks includes: The rendering information is divided into blocks according to a preset block division method to obtain the multiple information blocks; The block segmentation method refers to the way the encoder divides the rendered image into multiple macroblocks.

13. The method according to claim 12, characterized in that, The separate transmission of each information block includes: The transmission order of each information block is determined according to a preset encoding order, wherein the encoding order is the encoding order of the encoder for the multiple macroblocks; According to the transmission order corresponding to each information block, each information block is transmitted sequentially.

14. The method according to claim 12 or 13, characterized in that, The separate transmission of each information block includes: Each information block is transmitted to the encoder separately; or, Each information block is transmitted to the analysis module, which analyzes each information block to determine the corresponding encoding optimization information and transmits the encoding optimization information to the encoder. The encoder then performs encoding optimization based on the encoding optimization information of the corresponding information block during the encoding process of each macroblock.

15. A graphics processor, characterized in that, Used to perform the data processing method as described in any one of claims 1 to 14.

16. A data processing system, comprising: Graphics processor and encoder, The graphics processor is used for image rendering and to acquire rendering information related to the rendered image. The rendering information includes information used to assist the execution of the rendering process, information generated during the rendering process, or information input to the graphics processor. The rendered image is transmitted to the encoder, and the rendered information is compressed based on the parameters required in the encoding optimization process of the encoder, and the compressed rendered information is transmitted to the encoder. The encoder is used to encode the rendered image and to optimize the encoding based on the compressed rendering information during the encoding process.

17. A data processing system, comprising: Graphics processor and encoder, The graphics processor is used for image rendering and to acquire rendering information related to the rendered image. The rendering information includes information used to assist the execution of the rendering process, information generated during the rendering process, or information input to the graphics processor. The rendered image is transmitted to the encoder, and the rendering information is divided into multiple information blocks, and each information block is transmitted to the encoder respectively; The encoder is used to divide the rendered image into multiple macroblocks, and after receiving an information block, to encode the macroblock corresponding to the received information block, and to perform encoding optimization based on the received information block during the encoding process of the macroblock. The graphics processor is specifically used to divide the rendering information into blocks according to a preset block division method to obtain the plurality of information blocks; wherein, the block division method is the way in which the encoder divides the rendered image into a plurality of macroblocks.

18. A data processing system, comprising: Graphics processor, analysis module, and encoder, The graphics processor is used for image rendering and to acquire rendering information related to the rendered image. The rendering information includes information used to assist the execution of the rendering process, information generated during the rendering process, or information input to the graphics processor. The rendered image is transmitted to the encoder, and the rendered information is compressed based on the parameters required in the encoding optimization process of the encoder, and the compressed rendered information is transmitted to the analysis module. The analysis module is used to analyze the compressed rendering information, determine the encoding optimization information, and transmit the encoding optimization information to the encoder. The encoder is used to encode the rendered image and to optimize the encoding based on the encoding optimization information during the encoding process.

19. A data processing system, comprising: Graphics processor, analysis module, and encoder, The graphics processor is used for image rendering and to acquire rendering information related to the rendered image. The rendering information includes information used to assist the execution of the rendering process, information generated during the rendering process, or information input to the graphics processor. The rendered image is transmitted to the encoder, and the rendering information is divided into multiple information blocks, and each information block is transmitted to the analysis module. The analysis module is used to analyze each information block, determine the encoding optimization information corresponding to each information block, and transmit the encoding optimization information corresponding to each information block to the encoder. The encoder is used to divide the rendered image into multiple macroblocks, and after receiving an information block, to encode the macroblock corresponding to the received information block, and to perform encoding optimization based on the encoding optimization information of the received information block during the encoding process of the macroblock. The graphics processor is specifically used to divide the rendering information into blocks according to a preset block division method to obtain the plurality of information blocks; wherein, the block division method is the way in which the encoder divides the rendered image into a plurality of macroblocks.

20. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is coupled to the processor; The memory stores program instructions that, when executed by the processor, cause the processor to perform the data processing method as described in any one of claims 1 to 14.

21. A computer storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer or processor, causes the computer or processor to perform the data processing method as described in any one of claims 1 to 14.

Images