Image encoding and decoding method and device

Abstract

The embodiment of the invention provides an image encoding and decoding method and device. The image coding method provided by the embodiment of the invention comprises the following steps: acquiring an auxiliary image which is obtained based on a rendering intermediate result corresponding to a to-be-coded image; performing redundancy elimination processing on the to-be-coded image according to the auxiliary image to obtain a redundancy-eliminated image; and encoding the redundancy-removed image to obtain a code stream. According to the embodiment of the invention, the coding and decoding performance can be improved.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to an image encoding and decoding method and apparatus. Background Technology

[0002] Many edge-cloud collaborative scenarios (such as cloud gaming, 3D cloud conferencing, and collaborative editing of 3D scenes) require rendering to generate more realistic visuals and enhance the user experience. Rendering demands significant computing power, but edge devices, limited by physical constraints such as size and power consumption, often have far weaker computing power than cloud servers. Therefore, rendering is typically deployed on cloud servers, where the rendered images / videos are compressed and sent to the edge devices for display. As demands for rendering quality and the resolution of display devices continue to improve, the image quality and resolution of rendered images / videos also increase, leading to a substantial increase in bandwidth requirements for transmitting rendered images / videos.

[0003] Therefore, reducing the bandwidth required for transmitting and rendering images / videos and improving encoding and decoding performance is an important research direction. Summary of the Invention

[0004] This application provides an image encoding / decoding method and apparatus to improve encoding / decoding performance.

[0005] In a first aspect, this application provides an image encoding method, comprising: acquiring an auxiliary image, the auxiliary image being obtained based on a rendering intermediate result corresponding to an image to be encoded; performing redundancy removal processing on the image to be encoded according to the auxiliary image to obtain a deredundancy image; and encoding the deredundancy image to obtain a bitstream.

[0006] In this embodiment of the application, by using the intermediate rendering result as an auxiliary image or by preprocessing the intermediate rendering result to obtain an auxiliary image, since the auxiliary image has a high similarity to the image to be encoded, the image to be encoded can be deredundantly processed based on the auxiliary image, which can reduce more redundant information in the image to be encoded and improve encoding performance.

[0007] Rendering intermediate results refer to the intermediate data generated by the graphics rendering engine during the process of generating the image to be displayed (i.e., the rendered image) / video (i.e., the rendered video). The graphics rendering engine can perform rendering processing on a 3D scene to obtain the rendered image (i.e., the image to be encoded, also known as the rendering result) and rendering intermediate results. For example, intermediate rendering results may include, but are not limited to, computer graphics motion vectors (CGMV), intermediate rendering images (intermediate rendering images refer to images generated before generating the final rendered image (i.e., the rendered image mentioned above), the computational complexity of intermediate rendering images is lower than that of rendered images; among them, intermediate rendering images may be such as intermediate rendering images without indirect lighting rendering, intermediate rendering images without specular reflection processing, intermediate rendering images without specular highlight processing, etc.), position maps, normal maps, albedo maps, specular intensity maps, depth maps, ambient occlusion maps, mesh IDs, material IDs (where each material map corresponds to a material ID), render IDs (each object (or a 3D object model) corresponds to a render ID), depth information, etc., and this application embodiment does not specifically limit them.

[0008] The auxiliary image is obtained based on the intermediate rendering result corresponding to the image to be encoded. For example, the intermediate rendering result can be used as the auxiliary image or the intermediate rendering result can be preprocessed to obtain the auxiliary image. As mentioned above, the intermediate rendering result can be at least one of the following: albedo map, normal map, low-quality rendering map, position map, specular intensity map, or graphic motion vector. Therefore, the auxiliary image can include one or more of the aforementioned maps, or the auxiliary image can include one or more of the aforementioned maps after preprocessing.

[0009] The server can process each image frame sequentially according to the order of the image sequence in the video, and the image to be encoded can refer to the image frame that the server is currently processing.

[0010] The image to be encoded and the auxiliary image can refer to the entire frame or a partial image of a frame, such as an image patch or a region of interest (ROI). The size and shape of the partial image are not specifically limited.

[0011] In one embodiment, the server can perform deredundancy processing based on the auxiliary image and the image to be encoded to obtain a deredundancy image.

[0012] The redundancy removal process used by the server may include:

[0013] (1) When there is only one auxiliary image, calculate the residual between the auxiliary image and the image to be encoded (for example, the difference of the pixel values ​​at corresponding positions), that is, calculate the difference of the pixel values ​​at corresponding positions in the auxiliary image and the image to be encoded, so as to form a deredundant image.

[0014] (2) When there are multiple auxiliary images, calculate the residual between one of the auxiliary images and the image to be encoded, that is, calculate the difference of pixel values ​​at corresponding positions in the auxiliary image and the image to be encoded, and then calculate the difference between the pixel values ​​in another auxiliary image and the pixel values ​​at the corresponding positions mentioned above, and so on, until all the auxiliary images are calculated, and finally the pixel differences are used to form the deredundant image.

[0015] In another embodiment, the server can first obtain an adjusted auxiliary image, which is closer to the image to be encoded than a standard auxiliary image (i.e., the adjusted auxiliary image has a higher similarity to the image to be encoded); then, redundancy removal processing is performed on the adjusted auxiliary image and the image to be encoded to obtain a de-redundant image. Because the adjusted auxiliary image has a higher similarity to the image to be encoded, performing redundancy removal processing on the image to be encoded based on the adjusted auxiliary image can further reduce redundant information in the image to be encoded, resulting in better redundancy removal and thus further improving encoding performance.

[0016] In one embodiment, the adjustment method used by the server for the auxiliary image may include: obtaining at least one set of adjustment parameters based on the auxiliary image and the image to be encoded, wherein one or more pixel values ​​in the auxiliary image correspond to one set of adjustment parameters in the at least one set of adjustment parameters; constructing at least one adjustment function based on the at least one set of adjustment parameters; and adjusting the pixels of the corresponding auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image.

[0017] In one embodiment, the adjustment method used by the server for the auxiliary image may include: dividing the auxiliary image into blocks to obtain multiple auxiliary image blocks; for any auxiliary image block, obtaining a set of adjustment parameters corresponding to the corresponding block in the image to be encoded, wherein the set of adjustment parameters corresponds to one or more pixel values ​​in the auxiliary image block; constructing at least one adjustment function based on at least one set of adjustment parameters; and adjusting the pixels of the corresponding auxiliary image blocks according to the at least one adjustment function to obtain adjusted auxiliary image blocks, wherein multiple adjusted auxiliary image blocks constitute an adjusted auxiliary image.

[0018] The server can perform the aforementioned adjustments on the entire auxiliary image frame, or, as described above, first divide the auxiliary image into multiple auxiliary image blocks, and then perform the aforementioned adjustments on each auxiliary image block separately to obtain the adjusted auxiliary image. The block division mode used by the server to divide the auxiliary image can include: regular block division or irregular block division. Regular block division can, for example, divide the auxiliary image into 9 blocks, 16 blocks, etc., while irregular block division can be based on the geometric boundaries of the content in the auxiliary image. The aforementioned block division modes can be encoded into the bitstream.

[0019] In one embodiment, the server can divide the image into blocks based on object edges. For example, using the Geometric Partitioning Mode (GPM) algorithm, the image can be divided into "triangular + trapezoidal" regions. This way, the terminal device only needs to know the angle and offset of the dividing lines to successfully decode the image. It is evident that the GPM algorithm can segment the image and provide more detailed adjustment processing. Based on this, the server can write block information (including the correspondence between auxiliary image blocks and adjustment parameters) into the bitstream. This allows the terminal device to locate and acquire auxiliary image blocks based on the block information, and then adjust these auxiliary image blocks based on the corresponding adjustment parameters, resulting in better and more detailed adjustment effects and improved encoding performance.

[0020] In one embodiment, the adjustment function constructed by the server may include any one or a combination of the following functions: a proportional function, a linear function, a polynomial function, a power function, or a function represented by a neural network. For example, a power function (e.g., y = x). n Alternatively, a direct proportional function (e.g., y = kx), or a combination of a power function and a direct proportional function (e.g., y = kx). n Alternatively, a function represented by a neural network can be a function that takes a single pixel value as input and a single pixel value as output, with a neural network as an intermediate layer. It should be noted that the foregoing examples describe several forms of functions, but do not limit the adjustment function. The embodiments of this application do not specifically limit the type and composition of the adjustment function.

[0021] Based on the function to be constructed, the corresponding adjustment parameters of the function can be determined. For example, y = x n The corresponding transformation parameters are n and k, respectively. nThe corresponding transformation parameters are k and n. In this embodiment, the server can adjust the auxiliary image at the pixel level. That is, all pixel values ​​in the auxiliary image can be adjusted using the same type of adjustment function, but the adjustment parameters corresponding to each pixel value can be different. Alternatively, each pixel value in the auxiliary image can be adjusted using different types of adjustment functions, and the adjustment parameters corresponding to each pixel value can be different. For example, the auxiliary image includes 64 pixels, corresponding to 64 pixel values. These 64 pixel values ​​can correspond to 64 sets of adjustment parameters, meaning the adjustment parameters for each of the 64 pixel values ​​are different. Or, these 64 pixel values ​​can correspond to 2 sets of adjustment parameters, meaning some of the adjustment parameters for the 64 pixel values ​​are the same, while the adjustment parameters for another part of the pixel values ​​are different. Or, these 64 pixel values ​​can correspond to 3 or more sets (less than 64 sets) of adjustment parameters, meaning the 64 pixel values ​​are divided into different sets, the adjustment parameters for pixel values ​​in the same set are the same, while the adjustment parameters for pixel values ​​in different sets are different. In other words, any set of adjustment parameters can correspond to one or more pixel values ​​in the auxiliary image.

[0022] Once a set of adjustment parameters is determined, an adjustment function can be constructed based on this set of parameters. This adjustment function is used to adjust the pixels of the corresponding auxiliary image or auxiliary image block. The corresponding pixel value is input into the adjustment function, and the output of the adjustment function is regarded as the adjusted pixel value of the corresponding pixel. After all pixel values ​​in the auxiliary image have been adjusted according to the aforementioned process, all the adjusted pixel values ​​constitute the adjusted auxiliary image.

[0023] In one embodiment, the server can determine the adjustment parameters of a first pixel (which can be any pixel in the auxiliary image) in the auxiliary image by comparing bitstream sizes. For example, multiple sets of adjustment parameters can be preset, and multiple adjustment functions can be constructed. These multiple adjustment functions are then used to adjust the first pixel, resulting in multiple adjusted pixel values. The residuals between these multiple adjusted pixel values ​​and the corresponding pixel values ​​in the image to be encoded are calculated, yielding multiple residuals. These residuals are then encoded, and the bitstream sizes after encoding are measured. The adjustment parameter corresponding to the smallest bitstream is determined as the adjustment parameter of the first pixel.

[0024] In one possible implementation, the image boundaries in the rendered result may originate from the geometric boundaries of objects or boundaries created by lighting. On either side of these boundaries, the degree of matching between the auxiliary image and the rendered result may differ. For example, the relationship between the alpha image and the color of the rendered result is typically determined by the alpha image, lighting, and material properties (e.g., roughness / smoothness). Different objects on either side of the boundary may be spatially distant (e.g., object A occludes object B, but the spatial distance between objects A and B is significant), leading to differences in lighting. Under different lighting conditions, the mapping relationship between the alpha image and the rendered result is different. An alpha image multiplied by 0.1 might be very similar to the rendered result in detail in darker conditions, but multiplied by 2 might be needed in brighter conditions. Similarly, differences in the materials of the alpha image and the rendered result are also relevant. Therefore, the fitness function used for pixel values ​​near the boundaries should also be different.

[0025] By selecting appropriate adjustment functions and parameters for each pixel value in the auxiliary image, the auxiliary image of the region can be finely adjusted based on the feature matching degree of the region. This results in a higher similarity between the adjusted auxiliary image and the image to be encoded. Consequently, in the subsequent redundancy removal process, more redundant information between the auxiliary image and the image to be encoded can be reduced, thus improving coding performance.

[0026] In one embodiment, based on the above process, the server can transmit adjustment parameters for adjusting the auxiliary image to the terminal device. Therefore, the bitstream can also include at least one set of adjustment parameters. Furthermore, the bitstream can also include the correspondence between the aforementioned at least one set of adjustment parameters and the auxiliary image, auxiliary image block, or pixels in the auxiliary image or auxiliary image block. Optionally, the server can write adjustment parameter values ​​into the bitstream, and the terminal device can construct the final adjustment function based on a pre-agreed adjustment function and the aforementioned adjustment parameter values. Optionally, the server can write the constructed adjustment function into the bitstream, and then the terminal device can obtain the aforementioned adjustment function by decoding the bitstream. Optionally, the server can write an index related to the adjustment parameters and / or adjustment function into the bitstream, and then the terminal device can obtain the adjustment parameter or adjustment function corresponding to the index based on a pre-agreed correspondence, thereby obtaining the final adjustment function. It should be noted that the embodiments of this application can use various methods to transmit adjustment parameters and / or adjustment functions, and no specific limitation is made thereto.

[0027] As described above, the auxiliary images of the image to be encoded may include one or more. Therefore, the server can use the above method to adjust each auxiliary image to obtain one or more adjusted auxiliary images of the image to be encoded.

[0028] In one embodiment, the redundancy removal process employed by the server may include: when there is only one auxiliary image, calculating the residual between the adjusted auxiliary image and the image to be encoded (e.g., the difference in pixel values ​​at corresponding positions), that is, calculating the difference in pixel values ​​at corresponding positions in the adjusted auxiliary image and the image to be encoded, thereby forming a redundancy removal image.

[0029] In another embodiment, when there are multiple auxiliary images, the residual is calculated between one of the adjusted auxiliary images and the image to be encoded. That is, the difference between the pixel values ​​at corresponding positions in the adjusted auxiliary image and the image to be encoded is calculated. Then, the difference between the pixel value in another adjusted auxiliary image and the difference between the pixel values ​​at the corresponding positions is calculated. This process is repeated until all auxiliary images have been calculated. The resulting pixel differences form the deredundant image.

[0030] The server can encode the deredundant images using a standard video encoder to obtain a bitstream. This standard video encoder can be implemented according to industry video compression standards, such as ITU-T H.264, H.265 / HEVC, H.266 / VVC, AVS2, AVS3, or extensions of such standards. The technology in this application is not limited to any specific encoding standard.

[0031] In one possible implementation, the bitstream may also include CGMV. As mentioned above, CGMV is an intermediate rendering result. The redundancy removal process in this embodiment still retains spatial motion logic similar to the original image, thus enabling good cooperation with existing CGMV-assisted encoding / decoding algorithms.

[0032] In one possible implementation, the bitstream may also include intermediate rendering results or auxiliary images corresponding to the image to be encoded. The server can directly transmit the generated intermediate rendering results or auxiliary images to the terminal device, so that the terminal device can directly obtain the intermediate rendering results or auxiliary images from the bitstream without further rendering processing, thereby improving decoding performance. Alternatively, the server can adjust the auxiliary image and transmit it to the terminal device without transmitting the adjustment parameters. The terminal device can directly perform image reconstruction based on the adjusted auxiliary image, thereby improving the decoding efficiency of the terminal device.

[0033] Secondly, this application provides an image decoding method, comprising: receiving a bitstream; obtaining a deredundant image based on the bitstream; obtaining an auxiliary image; and obtaining a reconstructed image based on the auxiliary image and the deredundant image.

[0034] In this embodiment of the application, since the auxiliary image and the image to be encoded have a high similarity in the server, the redundant image obtained by performing redundancy removal processing on the image to be encoded based on the auxiliary image can reduce more redundant information in the image to be encoded. Encoding such a redundant image into a bitstream and transmitting it to the terminal device can achieve the effect of saving bitstream. The terminal device can obtain the reconstructed redundant image through the decoding stream, and reconstruct the reconstructed image by combining it with the auxiliary image, which can also improve decoding performance.

[0035] The terminal device can decode the bitstream using a standard video decoder to obtain a deredundant image. This standard video decoder can be implemented according to industry video compression standards, such as ITU-T H.264, H.265 / HEVC, H.266 / VVC, AVS2, AVS3, or extensions of such standards. The technology in this application is not limited to any specific decoding standard.

[0036] In one embodiment, the auxiliary image can be obtained based on intermediate rendering results (which may be obtained during the rendering process of the 3D scene by the terminal device, or from the decoded bitstream). The auxiliary image can be taken from the intermediate rendering results or a portion of the intermediate rendering results, or the auxiliary image can be obtained by preprocessing the intermediate rendering results or a portion of the intermediate rendering results.

[0037] In one embodiment, the auxiliary image may also be obtained from the decoded bitstream.

[0038] The deredundant image and auxiliary image can refer to the entire frame of the image or a local image of a frame, such as an image patch or a region of interest (ROI). The size and shape of the local image are not specifically limited.

[0039] In one embodiment, the terminal device may first acquire an adjusted auxiliary image, which is closer to the reconstructed image than the auxiliary image (i.e., the adjusted auxiliary image and the reconstructed image have a higher similarity); and then reconstruct the image based on the adjusted auxiliary image and the deduplicated image to obtain the reconstructed image.

[0040] In one embodiment, the bitstream further includes at least one set of adjustment parameters. Based on this, the adjustment method used by the terminal device to obtain the adjusted auxiliary image may include: obtaining at least one set of adjustment parameters based on the bitstream; constructing at least one adjustment function based on the at least one set of adjustment parameters; and adjusting the pixels of the corresponding auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image. Optionally, one or more pixels in the auxiliary image correspond to a set of adjustment parameters in the at least one set of adjustment parameters, and the pixels in the corresponding auxiliary image are adjusted according to the at least one adjustment function to obtain the adjusted auxiliary image.

[0041] In another embodiment, the bitstream further includes at least one set of adjustment parameters. Based on this, the adjustment method used by the terminal device to obtain the adjusted auxiliary image may include: obtaining at least one set of adjustment parameters based on the bitstream, wherein the auxiliary image includes multiple auxiliary image blocks, and each auxiliary image block corresponds to a set of adjustment parameters; constructing at least one adjustment function based on the at least one set of adjustment parameters; adjusting the pixels of the corresponding auxiliary image blocks according to the at least one adjustment function to obtain the adjusted auxiliary image block corresponding to the auxiliary image block; and obtaining the adjusted auxiliary image based on the multiple adjusted auxiliary image blocks. Optionally, a set of adjustment parameters corresponding to an auxiliary image block can be obtained based on the correspondence between auxiliary image blocks and adjustment parameters in the bitstream.

[0042] Optionally, the terminal device can also perform block processing on the auxiliary image. In one embodiment, the block processing method is determined according to the block processing mode, which can be preset or obtained from the bitstream. Specific block processing methods and block modes can be found in the server-side description above.

[0043] The terminal device adjusts the auxiliary image, resulting in an auxiliary image with higher similarity to the image to be encoded. This makes the reconstructed image, obtained by reconstructing from the deredundant image, closer to the image to be encoded. Furthermore, block-based processing saves computational resources at the decoding end.

[0044] The adjustment function on the decoding side can be referred to the description of the adjustment function on the encoding side above, and will not be repeated here.

[0045] Terminal devices can directly obtain adjustment parameters from the bitstream to construct corresponding adjustment functions. After verification by the server, such adjustment functions can perform detailed adjustments to the auxiliary image of the region based on the feature matching degree of the region. This results in the adjusted auxiliary image having a higher similarity to the image to be encoded. The terminal device can then reproduce the aforementioned adjustment function to achieve auxiliary image adjustment consistent with the server, thereby improving decoding performance.

[0046] As described above, the auxiliary images for the deredundancy image can include one or more. Therefore, the terminal device can use the above method to adjust each auxiliary image separately to obtain one or more adjusted auxiliary images for the deredundancy image.

[0047] In one embodiment, the reconstruction process employed by the terminal device may include: when there is only one auxiliary image, reconstructing based on the adjusted auxiliary image and the deduplication image (e.g., summing the pixel values ​​at corresponding positions), that is, calculating the sum of the pixel values ​​at corresponding positions in the adjusted auxiliary image and the deduplication image to form a reconstructed image.

[0048] When there are multiple auxiliary images, sum the adjusted auxiliary image and the deduplicated image. That is, calculate the sum of the pixel values ​​at corresponding positions in the adjusted auxiliary image and the deduplicated image. Then calculate the sum of the pixel values ​​in another adjusted auxiliary image and the sum of the pixel values ​​at the corresponding positions mentioned above. Continue in this way until all auxiliary images have been calculated. The final pixel sums form the reconstructed image.

[0049] In one embodiment, the terminal device may first perform frequency domain transformation on the auxiliary image to obtain an auxiliary image frequency map; perform frequency domain transformation on the deredundant image to obtain a deredundant image frequency map; then reconstruct the deredundant image frequency map based on the auxiliary image frequency map to obtain a reconstructed image frequency map; and finally perform inverse frequency domain transformation on the reconstructed image frequency map to obtain a reconstructed image.

[0050] In other words, the terminal device performs adjustments in the frequency domain, rather than applying different adjustment functions to different regions of the same image. Instead, it applies different adjustment functions to details at different frequencies of the image (performing block-based adjustments in the frequency domain is equivalent to adjusting by frequency level in the original image domain). Although the principle is different, performing adjustments in the frequency domain can still achieve redundancy removal by utilizing information from intermediate rendering results. Moreover, adjusting by frequency, since any frequency signal affects the pixel values ​​of the entire image, may result in an image with better spatial continuity, which can further improve encoding and decoding performance in some scenarios. Furthermore, by adjusting the auxiliary image by frequency, the resulting adjusted auxiliary image can have a higher similarity to the image to be encoded, thus making the reconstructed image more closely resemble the image to be encoded.

[0051] In one possible implementation, the bitstream also includes rendering intermediate results or auxiliary images. The terminal device directly decodes the stream to obtain the rendered intermediate results or auxiliary images without performing rendering processing, which can improve decoding performance. Alternatively, the terminal device directly decodes the stream to obtain the adjusted auxiliary images without performing rendering or adjusting the auxiliary images, which can improve the decoding efficiency of the terminal device.

[0052] Thirdly, this application provides an image encoding apparatus, comprising: a redundancy removal module for acquiring an auxiliary image, the auxiliary image being obtained based on a rendering intermediate result corresponding to the image to be encoded; performing redundancy removal processing on the image to be encoded according to the auxiliary image to obtain a redundancy-removed image; and an encoding module for encoding the redundancy-removed image to obtain a bitstream.

[0053] In one embodiment, the redundancy removal module is specifically used to obtain an adjusted auxiliary image, which is closer to the image to be encoded than the auxiliary image; and to perform redundancy removal processing based on the adjusted auxiliary image and the image to be encoded to obtain the redundancy removal image.

[0054] In one embodiment, the redundancy removal module is specifically configured to obtain at least one set of adjustment parameters based on the auxiliary image and the image to be encoded; construct at least one adjustment function based on the at least one set of adjustment parameters; and adjust the pixels of the corresponding auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image.

[0055] In one embodiment, the redundancy removal module is specifically used to divide the auxiliary image into blocks to obtain multiple auxiliary image blocks; for any auxiliary image block, obtain at least one set of adjustment parameters based on the auxiliary image block and the corresponding block in the image to be encoded; construct at least one adjustment function based on the at least one set of adjustment parameters; and adjust the pixels of the corresponding auxiliary image block according to the at least one adjustment function to obtain the adjusted auxiliary image block corresponding to the auxiliary image block.

[0056] In one embodiment, the bitstream further includes the at least one set of adjustment parameters.

[0057] In one embodiment, the adjustment function includes any one or a combination of the following functions: a proportional function, a linear function, a polynomial function, a power function, or a function represented by a neural network.

[0058] In one embodiment, the intermediate rendering result includes at least one of an albedo map, a normal map, a low-quality rendering map, or a position map.

[0059] In one embodiment, the bitstream may further include intermediate rendering results corresponding to the image to be encoded or the auxiliary image.

[0060] In one embodiment, the bitstream further includes block information, which includes the correspondence between the at least one set of adjustment parameters and the plurality of auxiliary image blocks.

[0061] In one embodiment, the redundancy removal module is further configured to perform frequency domain transformation on the auxiliary image to obtain an auxiliary image frequency map; perform frequency domain transformation on the image to be encoded to obtain an image frequency map to be encoded; perform redundancy removal processing on the image frequency map to be encoded based on the auxiliary image frequency map to obtain a redundancy-removed image frequency map; and perform inverse frequency domain transformation on the redundancy-removed image frequency map to obtain the redundancy-removed image.

[0062] Fourthly, this application provides an image decoding apparatus, comprising: a receiving module for receiving a bitstream; a decoding module for obtaining a deredundant image based on the bitstream; a reconstruction module for obtaining an auxiliary image; and obtaining a reconstructed image based on the auxiliary image and the deredundant image.

[0063] In one embodiment, the reconstruction module is specifically used to acquire an adjusted auxiliary image, which is closer to the reconstructed image than the auxiliary image; and to reconstruct the reconstructed image based on the adjusted auxiliary image and the deredundant image.

[0064] In one embodiment, the bitstream further includes at least one set of adjustment parameters; the reconstruction module is specifically configured to obtain at least one set of adjustment parameters based on the bitstream; construct at least one adjustment function based on the at least one set of adjustment parameters; and adjust the pixels of the corresponding auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image.

[0065] In one embodiment, the bitstream further includes at least one set of adjustment parameters; the reconstruction module is specifically configured to obtain at least one set of adjustment parameters based on the bitstream, wherein any set of adjustment parameters corresponds to an auxiliary image block, and the auxiliary image includes multiple auxiliary image blocks; construct at least one adjustment function based on the at least one set of adjustment parameters; adjust the pixels of the corresponding auxiliary image blocks according to the at least one adjustment function to obtain an adjusted auxiliary image block corresponding to the auxiliary image block; and obtain the adjusted auxiliary image based on the multiple adjusted auxiliary image blocks.

[0066] In one embodiment, the adjustment function includes any one or a combination of the following functions: a proportional function, a linear function, a polynomial function, a power function, or a function represented by a neural network.

[0067] In one embodiment, the bitstream further includes the intermediate rendering result corresponding to the deredundancy image or the auxiliary image.

[0068] In one embodiment, the auxiliary image is obtained from the rendering intermediates of the 3D scene based on the decoding end; or, the auxiliary image is obtained based on the bitstream.

[0069] In one embodiment, the intermediate rendering result includes at least one of an albedo map, a normal map, a low-quality rendering map, or a position map.

[0070] In one embodiment, the bitstream further includes block information, which includes the correspondence between the at least one set of adjustment parameters and the plurality of auxiliary image blocks.

[0071] In one embodiment, the reconstruction module is further configured to perform frequency domain transformation on the auxiliary image to obtain an auxiliary image frequency map; perform frequency domain transformation on the deredundant image to obtain a deredundant image frequency map; reconstruct the deredundant image frequency map based on the auxiliary image frequency map to obtain a reconstructed image frequency map; and perform inverse frequency domain transformation on the reconstructed image frequency map to obtain the reconstructed image.

[0072] Fifthly, this application provides an edge-cloud collaborative system, comprising: a server and a terminal device; wherein, the server is used for rendering a 3D scene to obtain an image to be encoded; selecting a first auxiliary image based on intermediate rendering results generated during the rendering process; performing redundancy removal processing on the image to be encoded based on the first auxiliary image to obtain a deredundant image; encoding the deredundant image and writing the encoded data of the deredundant image into a bitstream; sending the bitstream to the terminal device; the terminal device is used for receiving the bitstream; decoding the bitstream to obtain a decoding result, the decoding result including the deredundant image; rendering a 3D scene based on rendering parameters, selecting a second auxiliary image based on intermediate rendering results generated during the rendering process; and reconstructing based on the deredundant image and the second auxiliary image to obtain a reconstructed image.

[0073] In one embodiment, the resolution of the first auxiliary image is different from the resolution of the second auxiliary image; and / or, the content of the first auxiliary image is different from the content of the second auxiliary image.

[0074] In one embodiment, the resolution of the second auxiliary image is higher than that of the first auxiliary image.

[0075] In a sixth aspect, this application provides an apparatus comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in any one of the first or second aspects above.

[0076] In a seventh aspect, this application provides a computer-readable storage medium including a computer program that, when executed on a device, causes the device to perform the method described in any one of the first or second aspects above.

[0077] Eighthly, this application provides a computer program that, when executed by a device, performs the method described in any one of the first to second aspects.

[0078] Ninthly, this application provides a computer program product comprising computer program code, which, when executed on a device, causes the device to perform the method described in any one of the first to second aspects.

[0079] In a tenth aspect, this application provides a bitstream generated according to the method of the first aspect or any one of the first aspects described above.

[0080] Eleventhly, this application provides a computer-readable storage medium storing a bitstream generated according to the method described in the first aspect or any one of the first aspects.

[0081] In a twelfth aspect, an apparatus for storing a bitstream is provided, the apparatus comprising: a receiver and at least one storage medium, the receiver being configured to receive a bitstream generated according to the method described in the first aspect above, and the at least one storage medium being configured to store the bitstream.

[0082] In a thirteenth aspect, an apparatus for transmitting a bitstream is provided, the apparatus comprising: a transmitter and a receiver, the receiver being configured to receive a bitstream generated according to the method described in the first aspect above, and the transmitter being configured to transmit the bitstream to an end-side device via a transmission medium.

[0083] In a fourteenth aspect, an apparatus for transmitting a bitstream is provided, the apparatus comprising: a transmitter and at least one storage medium, the at least one storage medium being configured to store a bitstream generated according to the method described in the first aspect above, the transmitter being configured to retrieve the bitstream from the storage medium and transmit the bitstream to an end-side device via a transmission medium.

[0084] In a fifteenth aspect, a system for distributing bitstreams is provided, the system comprising: at least one storage medium for storing a bitstream generated according to the method described in the first aspect above; and a streaming media device for retrieving the bitstream from the at least one storage medium and sending the bitstream to an end-side device, wherein the streaming media device includes a content server or a content distribution server. Attached Figure Description

[0085] Figure 1aThis is an exemplary schematic diagram illustrating an application scenario of this application embodiment;

[0086] Figure 1b This is a schematic diagram illustrating an exemplary end-to-cloud collaborative system according to an embodiment of this application;

[0087] Figure 2a This is an exemplary schematic diagram illustrating the encoding process in an embodiment of this application;

[0088] Figure 2b This is a schematic diagram illustrating an exemplary encoding framework for an embodiment of this application;

[0089] Figure 3a This is an exemplary schematic diagram illustrating the decoding process in an embodiment of this application;

[0090] Figure 3b This is a schematic diagram illustrating an exemplary decoding framework for an embodiment of this application;

[0091] Figure 4 A flowchart of process 400 of an image encoding method provided in an embodiment of this application;

[0092] Figure 5 A schematic diagram of a geometric partitioning pattern provided in an embodiment of this application;

[0093] Figure 6 A flowchart of a process 600 for an image decoding method provided in an embodiment of this application;

[0094] Figure 7 A flowchart of an encoding end provided in an embodiment of this application;

[0095] Figure 8 A flowchart of a decoding end provided in an embodiment of this application;

[0096] Figure 9 A schematic diagram of an edge-cloud encoding / decoding architecture provided in an embodiment of this application;

[0097] Figure 10 A comparison diagram of the rendering results, Albedo diagram, and redundancy-removed rendering results provided in the embodiments of this application;

[0098] Figure 11 A schematic diagram of an edge-cloud encoding / decoding architecture provided in an embodiment of this application;

[0099] Figure 12 A schematic diagram of an edge-cloud encoding / decoding architecture provided in an embodiment of this application;

[0100] Figure 13 A schematic diagram of an edge-cloud encoding / decoding architecture provided in an embodiment of this application;

[0101] Figure 14This is a schematic diagram of the structure of an image encoding device 1400 provided in an embodiment of this application;

[0102] Figure 15 This is a schematic diagram of the structure of an image encoding device 1500 provided in an embodiment of this application. Detailed Implementation

[0103] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0104] The terms "first," "second," etc., used in the specification, embodiments, claims, and drawings of this application are for distinguishing purposes only and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0105] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0106] The following are keywords involved in the embodiments of this application:

[0107] Intermediate rendering results: Intermediate data generated by the graphics rendering engine during the process of generating the image / video to be displayed. This may include albedo maps, normal maps, low-quality rendering results, position maps, specular intensity maps, depth maps, ambient occlusion maps, graphic motion vectors, Mesh IDs, Material IDs, and one or more other types of information generated during the rendering process.

[0108] Rendering parameters: Input to the graphics rendering engine, including various rendering parameters, position and color vectors of all light sources, position vectors of the player or observer, sampling method of each texture, position coordinates of each scene object, motion trajectory of moving objects, and skeletal animation parameters.

[0109] Reference frame: A concept in video coding methods. It is generally similar to the frame to be encoded. Subtracting the frame to be encoded from the reference frame will result in an image with less information, which is more conducive to encoding.

[0110] Auxiliary images: Auxiliary images primarily include images from the intermediate rendering results. In this application, further preprocessing of the intermediate rendering results images is involved, and the resulting images serve as input to subsequent adjustment modules. The intermediate rendering results and the images generated through the preprocessing steps of this application are collectively referred to as auxiliary images. Auxiliary images often have similar visual features to the final rendering result but cannot be directly used as reference frames. This application utilizes these auxiliary images to remove redundant information from the image to be encoded. The aforementioned preprocessing may include any one or more of anti-aliasing, feature extraction, normal processing, or high-pass filtering.

[0111] Adjustment: Each pixel in the auxiliary image is input into a function, and the output value of the function is used as the pixel value of the new image. The adjusted image can be subtracted from the frame to be encoded, just like the reference frame, to produce a de-redundant image with less information.

[0112] Adjustment parameters: These are parameters used to describe the specific adjustment operations and are the coefficients of the function used for adjustment.

[0113] Standard video encoders and standard video decoders: These can be implemented according to industry video compression standards, such as ITU-T H.264, H.265 / HEVC, H.266 / VVC, AVS2, AVS3, or extensions of such standards. The technology in this application is not limited to any particular encoding / decoding standard.

[0114] Figure 1a This is an exemplary schematic diagram illustrating an application scenario of an embodiment of this application. For example... Figure 1aAs shown, the embodiments of this application can be applied to various end-to-end cloud collaboration scenarios, such as cloud gaming, 3D cloud conferencing, and collaborative editing and production of 3D scenes, without specific limitations. This application uses cloud gaming as an example for illustration.

[0115] Reference Figure 1a In one embodiment, the device may include a mobile phone, a personal computer (PC), virtual reality (VR) glasses, augmented reality (AR) headset, tablet computer, and other terminal devices. These devices send the first rendering parameters to a central server (e.g., a cloud gaming server) via an edge server. The central server then renders the 3D scene based on the rendering parameters (which may include only the first rendering parameters uploaded by the terminal devices, or a combination of the first rendering parameters uploaded by the terminal devices and second rendering parameters generated by the central server) to obtain a rendered image (e.g., a game image). Subsequently, the central server obtains an auxiliary image based on the intermediate rendering results generated during the rendering process. It then performs deredundancy processing on the rendered image based on the auxiliary image to obtain a deredundancy image. The deredundancy image is encoded to obtain a bitstream, which is then distributed to each terminal device via the edge server. Subsequently, the terminal device can render the 3D scene based on rendering parameters (including a first rendering parameter and a fifth rendering parameter, the fifth rendering parameter being generated by the terminal device and potentially containing the same parameters as the second rendering parameter), and generate intermediate rendering results during the rendering process; then, the bitstream is decoded to obtain a deredundant image. The terminal device then acquires an auxiliary image based on the intermediate rendering results, and performs reconstruction based on the auxiliary image and the deredundant image to obtain the reconstructed image.

[0116] Reference Figure 1aIn one embodiment, the device may include a mobile phone, PC, VR glasses, AR headset, tablet computer, and other terminal devices, which send the first rendering parameters to the central server via an edge server. The central server then renders the 3D scene based on the rendering parameters (which may include only the first rendering parameters uploaded by the terminal devices, or include both the first rendering parameters uploaded by the terminal devices and the second rendering parameters generated by the central server) to obtain a rendered image. Next, the central server obtains an auxiliary image based on the intermediate rendering results generated during the rendering process, adjusts the auxiliary image to obtain an adjusted auxiliary image, and then performs redundancy removal processing on the rendered image based on the adjusted auxiliary image to obtain a deredundant image. The deredundant image is then encoded; and the adjustment parameters or adjustment functions used in the aforementioned adjustment process are encoded to obtain a bitstream, which is then distributed to each terminal device via the edge server. Subsequently, the terminal devices render the 3D scene based on the rendering parameters (which include the first and fifth rendering parameters), generate intermediate rendering results during the rendering process, decode the bitstream to obtain the deredundant image, and the adjustment parameters or adjustment functions used in subsequent adjustment processes. Then, the terminal device obtains an auxiliary image based on the intermediate rendering result, adjusts the auxiliary image to obtain an adjusted auxiliary image, and then reconstructs the image based on the adjusted auxiliary image and the deduplicated image.

[0117] Since the intermediate rendering results are highly correlated with the rendered image, this embodiment performs redundancy removal processing on the rendered image based on an auxiliary image (obtained from the intermediate rendering results). This reduces redundant information in the rendered image and improves encoding performance. Furthermore, the amount of data in the rendering parameters is much smaller than the amount of data in the intermediate rendering results. Therefore, even if the central server also sends rendering parameters to the terminal device, the bitrate overhead of the data stream transmitted from the central server to the terminal device is less than the bitrate overhead of the data stream transmitted in the prior art.

[0118] For example, rendering parameters can refer to all parameters input to the graphics rendering engine and required for rendering processing by the graphics rendering engine. These can include various rendering parameters, the position and color vectors of all light sources, the position vectors of the player or observer, the sampling method of each texture, the position coordinates of each scene object, the motion trajectory of moving objects, skeletal animation parameters, etc. This application embodiment does not specifically limit these parameters. The first rendering parameter and the second rendering parameter can constitute the rendering parameters (i.e., all parameters input to the graphics rendering engine and required for rendering by the graphics rendering engine). It should be noted that the types of parameters included in the fifth rendering parameter generated by the terminal device are the same as the types of parameters included in the second rendering parameter generated by the central server, and the precision of the parameters included in the fifth rendering parameter is less than or equal to the precision of the parameters included in the second rendering parameter.

[0119] For example, intermediate rendering results are a subset of all intermediate rendering results generated during the rendering process. It should be noted that the types of intermediate results included in the intermediate rendering results generated by the terminal device are the same as those included in the intermediate rendering results generated by the central server. The precision of the intermediate results included in the intermediate rendering results generated by the terminal device may be the same as or different from that of the intermediate rendering results generated by the central server; for example, the former may be higher than or equal to the latter. Furthermore, the textures of the intermediate results included in the intermediate rendering results generated by the terminal device may also be the same as or different from those of the intermediate rendering results generated by the central server; no specific limitations are imposed on this.

[0120] based on Figure 1a In this application scenario, an end-to-cloud collaborative system is proposed, which can be used as follows: Figure 1b As shown.

[0121] Figure 1b This is a schematic diagram illustrating an exemplary end-to-cloud collaborative system according to an embodiment of this application. Figure 1b As shown, the edge-cloud collaborative system can include servers and terminal devices.

[0122] Reference Figure 1b For example, a server can refer to Figure 1a The central server in the process can be a single server or a server cluster; this application does not specifically limit this.

[0123] For example, the server may include a first rendering module, an encoder, and a first communication module. It should be understood that... Figure 1b This is merely one example of an embodiment of this application; the server in this embodiment may include, but is not limited to, a different type of server. Figure 1b The embodiments of this application do not specifically limit the number of more or fewer modules shown.

[0124] For example, the first rendering module can be used to render a 3D scene to obtain a rendered image and intermediate rendering results during the rendering process;

[0125] It should be understood that, in the embodiments of this application, the rendered image described on the server, cloud side, or encoding end refers to the image to be encoded.

[0126] For example, the first rendering module can be used to obtain an auxiliary image based on the intermediate rendering result, and to perform redundancy removal processing on the rendered image based on the auxiliary image to obtain a deredundant image.

[0127] For example, an encoder can be used to encode a deredundant image to obtain a bitstream.

[0128] For example, the first communication module can be used to communicate with other electronic devices, such as sending the code stream output by the encoder to a terminal device.

[0129] Reference Figure 1b For example, terminal devices include, but are not limited to: personal computers, computer workstations, smartphones, tablets, servers, smart cameras, smart cars or other types of cellular phones, media consumption devices, wearable devices (such as VR / AR headsets, VR glasses), set-top boxes, game consoles, etc.

[0130] For example, the terminal device may include a second communication module, a second rendering module, a decoder, and a display module. It should be understood that... Figure 1b This is merely one example of an embodiment of this application; the terminal device in this embodiment may include, but is not limited to, a more advanced device. Figure 1b The embodiments of this application do not specifically limit the number of more or fewer modules shown.

[0131] For example, the second communication module can be used to communicate with other electronic devices, such as receiving a bitstream sent by a server. For example, the second rendering module can be used to render a 3D scene to obtain intermediate rendering results. It should be understood that the second rendering module can also generate rendered images.

[0132] For example, the second rendering module can be used to obtain an auxiliary image based on intermediate rendering results.

[0133] For example, a decoder can be used to decode a bitstream to obtain a deredundant image, and the decoder can also be used to obtain a reconstructed image based on the auxiliary image and the deredundant image.

[0134] For example, a display module can be used to display a reconstructed image.

[0135] It should be understood that the embodiments of this application do not limit the video coding standards used by the encoder and decoder, and may include, but are not limited to: H.264 / Advanced Video Coding (AVC), H.265 / High Efficiency Video Coding (HEVC), H.266 / Versatile Video Coding (VVC), AV1 (AOMedia Video 1, "AOMedia" refers to a video coding standard developed by the Open Media Video Consortium), and extension standards of these video coding standards; in addition, it may also include new video coding standards and extension standards that have emerged with the development of video coding and decoding technologies.

[0136] The following is based on Figure 1a and Figure 1b The encoding and decoding process is explained.

[0137] Figure 2a This is an exemplary schematic diagram illustrating the encoding process of an embodiment of this application. Figure 2a In this embodiment, the server neither sends the first intermediate rendering result to the terminal device nor sends the second rendering parameters to the terminal device. This application embodiment uses encoding a frame of rendered image as an example for illustration.

[0138] S201, the server renders the 3D scene to obtain a rendered image.

[0139] The rendering parameters used in the rendering process include rendering parameters obtained from the terminal device, rendering parameters stored locally, or pre-set rendering parameters.

[0140] For example, the server can obtain rendering parameters; then, these parameters can be input into the graphics rendering engine, which then renders the 3D scene based on the parameters to obtain a rendered image. The graphics rendering engine belongs to... Figure 1b The first rendering module in the process.

[0141] For example, the server can obtain first rendering parameters from the terminal device (wherein the first rendering parameters are generated by the terminal device). In one embodiment, when the first rendering parameters generated by the terminal device are all the parameters required by the graphics rendering engine for rendering processing, the server can determine the first rendering parameters as the rendering parameters; that is, the rendering parameters obtained by the server may only include the first rendering parameters. In one embodiment, when the first rendering parameters generated by the terminal device are only some of the parameters required by the graphics rendering engine for rendering processing, the server can also generate second rendering parameters; then, the first rendering parameters and the second rendering parameters are determined as the rendering parameters. That is, the rendering parameters obtained by the server may include both the first rendering parameters and the second rendering parameters. The first rendering parameters and the second rendering parameters will be described in detail in subsequent embodiments.

[0142] It should be noted that the server can obtain the first rendering parameters from the terminal device according to a preset period. The preset period can be set as needed, and this embodiment does not specifically limit it. It should be understood that the first rendering parameters obtained by the server from the terminal device each time can be used to render one or more frames of image.

[0143] For example, the rendering process of a graphics rendering engine can be as follows: the graphics engine can perform geometric transformations, projection transformations, perspective transformations, and window clippings on the three-dimensional object models in the three-dimensional scene based on the input rendering parameters, and then generate a rendered image by using the acquired material and lighting information.

[0144] S202, the server obtains an auxiliary image based on the intermediate rendering results generated during the rendering process.

[0145] For example, the graphics rendering engine in the first rendering module will generate intermediate rendering results during the rendering process (the rendering process refers to the time period between the moment when the rendering parameters are obtained from the image rendering engine and the moment when the rendered image is obtained).

[0146] Optionally, in one embodiment, a first intermediate rendering result can be selected from the intermediate rendering results. For example, a portion of the intermediate rendering results can be selected as the first intermediate rendering result.

[0147] An auxiliary image is obtained based on the intermediate rendering result or the first intermediate rendering result. In one embodiment, the auxiliary image may be the intermediate rendering result or a partial image of the intermediate rendering result. In another embodiment, the auxiliary image may be the first intermediate rendering result. In yet another embodiment, the intermediate rendering result, a partial image of the intermediate rendering result, or the first intermediate rendering result may be preprocessed to obtain the auxiliary image. The preprocessing in this embodiment may include, but is not limited to, one or more of the following: anti-aliasing processing, feature extraction, high-pass filtering, normal processing, etc.

[0148] It should be noted that S201 to S202 can be... Figure 1b The first rendering module in the process is executed, but this embodiment does not specifically limit this. The following S203-S204 can be performed by... Figure 1b The encoder in the code is executed.

[0149] S203, the server performs redundancy removal on the rendered image based on the auxiliary image to obtain a deredundant image.

[0150] S204, the server encodes the deredundancy image and writes the encoded data of the deredundancy image into the bitstream.

[0151] Figure 2b This is a schematic diagram illustrating an exemplary encoding framework for an embodiment of this application. Figure 2b The embodiments are shown Figure 1b The encoding framework of the encoder. Figure 2b The video to be encoded is a deredundant video, which may include multiple deredundant images.

[0152] Reference Figure 2b For example, the encoder may include: a block module, an intra-frame prediction module, an inter-frame prediction module, a transform module, a quantization module, an entropy coding module, an inverse quantization module, an inverse transform module, a loop filtering module, and a memory.

[0153] The encoding process of the encoder can be as follows: After the encoder's block segmentation module obtains the video to be encoded, it can segment each frame of the video to be encoded into blocks by removing redundant images, thus obtaining blocks to be encoded. For each block to be encoded, the decision module (…) Figure 2b (Not shown in the image) Determine whether to perform inter-frame prediction or intra-frame prediction on the block to be coded.

[0154] When intra-frame prediction is determined, the intra-frame prediction module performs intra-frame prediction to determine the prediction block corresponding to the block to be coded. After obtaining the prediction block, the residual block between the block to be coded and the prediction block can be determined. Then, the residual block is input to the transform module, which transforms the residual block, obtains the transform result, and outputs the transform result to the quantization module. Subsequently, the quantization module quantizes the transform result, obtains the quantization result, and outputs the quantization result to the entropy coding module. Then, the entropy coding module entropy codes the quantization result to obtain the encoded data of the residual block, and writes the encoded data of the residual block into the bitstream.

[0155] For example, the quantization module can also output the quantization result to the inverse quantization module, which performs inverse quantization to obtain the inverse quantization result and outputs it to the inverse transform module. Next, the inverse transform module performs an inverse transform on the inverse quantization result to obtain a decoded residual block. Then, the decoded residual block can be superimposed with the prediction block output by the intra-frame prediction module to obtain a reconstructed block. Subsequently, the reconstructed block is input to the loop filtering module, which performs loop filtering on the reconstructed block to obtain a filtered reconstructed block and outputs it to the memory. (The loop filtering module, after obtaining the filtered reconstructed block, can stitch the filtered reconstructed block to the corresponding position in the image to be reconstructed; after obtaining a frame of reconstructed image, it can output the reconstructed image to the memory for storage.)

[0156] When it is determined to be inter-frame prediction, the inter-frame prediction module can perform inter-frame prediction to determine the prediction block corresponding to the block to be encoded.

[0157] In one embodiment, the inter-frame prediction module can select a reference frame from a list of reference frames in memory as the reference frame for the deredundancy image. For the block to be encoded in the deredundancy image, a matching prediction block can be found from the reference frame.

[0158] In one embodiment, the inter-frame prediction module can select original reference frames from a list of reference frames in memory as candidate reference frames for the deredundant image; where the original reference frame refers to the reconstructed image. For the block to be encoded in the deredundant image, inter-frame prediction can be performed based on multiple candidate reference frames to determine multiple prediction blocks; where one candidate reference frame corresponds to one prediction block; then, an optimal prediction block can be selected from the multiple prediction blocks (for example, the prediction block with the lowest rate-distortion cost can be determined as the optimal prediction block using rate-distortion cost as the evaluation criterion).

[0159] After obtaining the prediction block, the residual block between the block to be encoded and the prediction block can be determined. Next, the residual block is input to the transform module, which transforms the residual block to obtain the transform result and outputs it to the quantization module. Subsequently, the quantization module quantizes the transform result to obtain the quantization result and outputs it to the entropy coding module. Then, the entropy coding module entropy codes the quantization result to obtain the encoded data of the residual block and writes the encoded data of the residual block into the bitstream (that is, the bitstream obtained in S204 above).

[0160] For example, the quantization module can also output the quantization result to the inverse quantization module, which performs inverse quantization to obtain the inverse quantization result and outputs it to the inverse transform module. Next, the inverse transform module performs an inverse transform on the inverse quantization result to obtain a decoded residual block. Then, the decoded residual block can be superimposed with the prediction block output by the intra-frame prediction module to obtain a reconstructed block. Subsequently, the reconstructed block is input to the loop filtering module, which performs loop filtering on the reconstructed block to obtain a filtered reconstructed block and outputs it to the memory. (The loop filtering module, after obtaining the filtered reconstructed block, can stitch the filtered reconstructed block to the corresponding position in the image to be reconstructed; after obtaining a frame of reconstructed image, it can output the reconstructed image to the memory for storage.)

[0161] Figure 3a This is an exemplary schematic diagram illustrating the decoding process in an embodiment of this application. Figure 3a What is shown is with Figure 2a The corresponding decoding process. In Figure 3a In one embodiment, the terminal device performs rendering processing based on locally generated rendering parameters and generates a first intermediate rendering result. Figure 3a In this embodiment, decoding a single frame of image is used as an example for illustration.

[0162] S301, the terminal device receives the bit stream.

[0163] For example, after the server sends the bitstream, the terminal device can receive the bitstream.

[0164] S302, the terminal device decodes the bitstream to obtain the decoding result, which includes the deredundant image.

[0165] Figure 3b This is a schematic diagram of an exemplary decoding framework. Figure 3b The embodiments are shown Figure 1b The decoding framework of the decoder.

[0166] Reference Figure 3bFor example, the decoder may include: an entropy decoding module, an inverse quantization module, an inverse transform module, an intra-frame prediction module, an inter-frame prediction module, a loop filtering module, and a memory.

[0167] For example, after receiving the bitstream, the terminal device can decode the bitstream to obtain the decoding result. The decoding process can be as follows: The encoded data of the residual block is extracted from the bitstream. Then, the encoded data of the residual block is input to the entropy decoding module, which performs entropy decoding on the encoded data of the residual block to obtain entropy-decoded data. Next, the entropy-decoded data can be input to the dequantization module, which performs dequantization to obtain dequantized data. This dequantized data is then input to the inverse transform module. Subsequently, the inverse transform module performs an inverse transform on the dequantized data to obtain the decoded residual block.

[0168] It should be understood that when the encoding end performs lossy encoding on the residual block, the residual block obtained by the decoding end will differ from the residual block encoded by the encoding end. When the encoding end performs lossless encoding on the residual block, the residual block obtained by the decoding end will be the same as the residual block encoded by the encoding end.

[0169] It should be noted that bitstream description information can also be extracted from the bitstream. In this case, there is no need to perform entropy decoding, inverse quantization, or inverse transform operations on the bitstream description information. This bitstream description information can be used in the subsequent decoding process.

[0170] For example, when it is determined to be intra-frame prediction based on the bitstream description information decoded from the bitstream, the intra-frame prediction module can perform intra-frame prediction to obtain a prediction block. Then, the decoded residual block can be superimposed with the prediction block to obtain a reconstructed block. Next, a loop filter can be applied to the reconstructed block to obtain a filtered reconstructed block, which is then output to a memory. (The loop filter module, after obtaining the filtered reconstructed block, can stitch it to the corresponding position in the image to be reconstructed; after obtaining a reconstructed image frame, it can output the reconstructed image to a memory for storage.)

[0171] For example, when it is determined to be inter-frame prediction based on the bitstream description information decoded from the bitstream, the inter-frame prediction module can perform inter-frame prediction to obtain the prediction block.

[0172] In one embodiment, when the inter-frame prediction module determines that the reference frame is the original reference frame from the bitstream description information decoded from the bitstream, the inter-frame prediction module can select the corresponding original reference frame as the reference frame from the reference frame list in the memory; then, the prediction block can be determined based on the original reference frame.

[0173] After obtaining the prediction block, the decoded residual block can be superimposed with the prediction block to obtain the reconstructed block. Then, a loop filter can be applied to the reconstructed block to obtain a filtered reconstructed block, which is then output to the memory. (The loop filter module, after obtaining the filtered reconstructed block, can stitch the filtered reconstructed block to the corresponding position in the image to be reconstructed; after obtaining a frame of reconstructed image, the reconstructed image can be output to the memory.) Figure 3b The reconstructed image is a deredundant image, and multiple deredundant images are combined to form a deredundant video.

[0174] It should be understood that the terminal device can also generate a bitstream based on the first rendering parameters and then send it to the server according to a preset period. In this way, the server can obtain the first rendering parameters. In addition, since the data volume of the first rendering parameters is small, the terminal device can encode the first rendering parameters and write the encoded data of the first rendering parameters into the bitstream before sending it to the server; the terminal device can also choose not to encode the first rendering parameters, but directly write the first rendering parameters into the bitstream and send it to the server; the embodiments of this application do not specifically limit this.

[0175] S303: The terminal device renders a 3D scene, generates intermediate rendering results during the rendering process, and obtains auxiliary images based on the intermediate rendering results.

[0176] For example, after decoding to obtain the deredundant image, the second rendering module (which may be a graphics rendering engine contained in the second rendering module) can perform rendering processing on the 3D scene.

[0177] For example, the second rendering module can render the 3D scene based on rendering parameters. When the first rendering parameters uploaded by the terminal device to the server are all the parameters required for the graphics rendering engine to perform rendering, the second rendering module can render the 3D scene based on the first rendering parameters corresponding to the current frame. When the first rendering parameters uploaded by the terminal device are only some of the parameters required for the graphics rendering engine to perform rendering, the terminal device can also generate a fifth rendering parameter. Then, the second rendering module can render the 3D scene based on the first and fifth rendering parameters corresponding to the current frame.

[0178] It should be noted that the types of parameters included in the fifth rendering parameters generated by the terminal device are the same as the types of parameters included in the second rendering parameters generated by the central server; the precision of the parameters included in the fifth rendering parameters is less than or equal to the precision of the parameters included in the second rendering parameters.

[0179] Optionally, in one embodiment, a first intermediate rendering result can be selected from the intermediate rendering results. For example, a portion of the intermediate rendering results can be selected as the first intermediate rendering result.

[0180] An auxiliary image is obtained based on the intermediate rendering result or the first intermediate rendering result. In one embodiment, the auxiliary image may be the intermediate rendering result or a partial image of the intermediate rendering result. In another embodiment, the auxiliary image may be the first intermediate rendering result. In yet another embodiment, the intermediate rendering result, a partial image of the intermediate rendering result, or the first intermediate rendering result may be preprocessed to obtain the auxiliary image. The preprocessing in this embodiment may include, but is not limited to, one or more of the following: anti-aliasing processing, feature extraction, high-pass filtering, normal processing, etc.

[0181] It should be noted that the type of intermediate results included in the first intermediate rendering result generated by the terminal device is the same as the type of intermediate results included in the first intermediate rendering result generated by the server; the precision of the intermediate results included in the first intermediate rendering result generated by the terminal device may be the same as or different from the precision of the intermediate results included in the first intermediate rendering result generated by the server, for example, the former may be higher than or equal to the latter. Furthermore, the texture of the intermediate results included in the first intermediate rendering result generated by the terminal device may also be the same as or different from the texture of the intermediate results included in the first intermediate rendering result generated by the server, without specific limitations.

[0182] It should be noted that the graphics rendering engine of the terminal device can also produce a rendered image. However, since the computing power of the terminal device is lower than that of the server, the quality of the rendered image obtained by the graphics rendering engine of the terminal device (where image quality can include subjective quality and objective quality (objective quality such as rate distortion performance and PSNR)) is lower than that of the rendered image obtained by the graphics rendering engine of the server. Therefore, the terminal device may choose not to display the rendered image obtained by the graphics rendering engine of the terminal device, but instead display the reconstructed image obtained by decoding the bitstream.

[0183] It should be noted that S301 to S302 can be... Figure 1b The decoder in the process is executed, but this embodiment of the application does not specifically limit this. S303 to S304 below can be executed by the second rendering module in the terminal device.

[0184] S304, the terminal device reconstructs the image based on the auxiliary image and the deredundant image to obtain the reconstructed image.

[0185] Since the auxiliary image is highly correlated with the rendered image, the embodiments of this application encode the rendered image based on the auxiliary image, which can ensure the quality of image reconstruction.

[0186] Figure 4 A flowchart of process 400 of the image encoding method provided in this application embodiment. Process 400 can be executed by the server described above. Process 400 is described as a series of steps or operations, and it should be understood that process 400 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 4 The execution order is shown. Process 400 may include:

[0187] Step 401: Obtain an auxiliary image, which is obtained based on the intermediate rendering result corresponding to the image to be encoded.

[0188] The server can process each image frame sequentially according to the order of the image sequence in the video, and the image to be encoded can refer to the image frame that the server is currently processing.

[0189] Intermediate rendering results may include, but are not limited to: CGMV, intermediate rendering images, position map, normal map, albedo map, specular intensity map, depth map, ambient occlusion map, Mesh ID, Material ID, RenderID, depth information, etc.

[0190] The auxiliary image is obtained based on the intermediate rendering result corresponding to the image to be encoded. The auxiliary image can be taken from the intermediate rendering result or a portion of the intermediate rendering result, or the auxiliary image can be obtained by preprocessing the intermediate rendering result or a portion of the intermediate rendering result.

[0191] The image to be encoded and the auxiliary image can refer to the entire frame or a partial image of a frame, such as an image patch or a region of interest (ROI). The size and shape of the partial image are not specifically limited.

[0192] Step 402: Perform redundancy removal processing on the image to be encoded based on the auxiliary image to obtain a redundancy-removed image.

[0193] In one embodiment, the server can perform deredundancy processing based on the auxiliary image and the image to be encoded to obtain a deredundancy image.

[0194] The redundancy removal process used by the server may include:

[0195] (1) When there is only one auxiliary image, calculate the residual between the auxiliary image and the image to be encoded (for example, the difference of the pixel values ​​at corresponding positions), that is, calculate the difference of the pixel values ​​at corresponding positions in the auxiliary image and the image to be encoded, so as to form a deredundant image.

[0196] (2) When there are multiple auxiliary images, calculate the residual between one of the auxiliary images and the image to be encoded, that is, calculate the difference of pixel values ​​at corresponding positions in the auxiliary image and the image to be encoded, and then calculate the difference between the pixel values ​​in another auxiliary image and the pixel values ​​at the corresponding positions mentioned above, and so on, until all the auxiliary images are calculated, and finally the pixel differences are used to form the deredundant image.

[0197] In another embodiment, the server can first obtain an adjusted auxiliary image, which is closer to the image to be encoded than a standard auxiliary image (i.e., the adjusted auxiliary image has a higher similarity to the image to be encoded); then, redundancy removal processing is performed on the adjusted auxiliary image and the image to be encoded to obtain a de-redundant image. Because the adjusted auxiliary image has a higher similarity to the image to be encoded, performing redundancy removal processing on the image to be encoded based on the adjusted auxiliary image can further reduce redundant information in the image to be encoded, resulting in better redundancy removal and thus further improving encoding performance.

[0198] In one embodiment, the adjustment method used by the server for the auxiliary image may include: obtaining at least one set of adjustment parameters based on the auxiliary image and the image to be encoded; constructing at least one adjustment function based on the at least one set of adjustment parameters; and adjusting the pixels of the corresponding auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image.

[0199] In another embodiment, the adjustment method used by the server for the auxiliary image may include: dividing the auxiliary image into blocks to obtain multiple auxiliary image blocks; for any auxiliary image block, obtaining at least one set of adjustment parameters based on the auxiliary image block and the corresponding block in the image to be encoded; constructing at least one adjustment function based on the at least one set of adjustment parameters; and adjusting the pixels of the corresponding auxiliary image blocks according to the at least one adjustment function to obtain adjusted auxiliary image blocks, wherein multiple adjusted auxiliary image blocks constitute an adjusted auxiliary image.

[0200] The server can perform the aforementioned adjustments on the entire auxiliary image frame, or, as described above, first divide the auxiliary image into multiple auxiliary image blocks, and then perform the aforementioned adjustments on each auxiliary image block separately to obtain the adjusted auxiliary image. The block division mode used by the server to divide the auxiliary image can include: regular block division or irregular block division. Regular block division can, for example, divide the auxiliary image into 9 blocks, 16 blocks, etc., while irregular block division can be based on the geometric boundaries of the content in the auxiliary image. The aforementioned block division modes can be encoded into the bitstream.

[0201] In one embodiment, the server can divide the object into blocks based on its edges, for example, such as... Figure 5 ( Figure 5 As shown in the diagram (Geometric Partitioning Mode), the Geometric Partitioning Mode (GPM) algorithm divides the image into "triangular + trapezoidal" regions. This allows the terminal device to successfully decode the image simply by knowing the angle and offset of the dividing lines. It is evident that the GPM algorithm can segment the image and provide more detailed adjustment processing. Based on this, the server can write segmentation information (including the correspondence between auxiliary image blocks and adjustment parameters) into the bitstream. This allows the terminal device to locate and acquire auxiliary image blocks based on the segmentation information, and then adjust these auxiliary image blocks according to the corresponding adjustment parameters, resulting in better and more detailed adjustment effects and improved encoding performance.

[0202] In one embodiment, the adjustment function constructed by the server may include any one or a combination of the following functions: a proportional function, a linear function, a polynomial function, a power function, or a function represented by a neural network. For example, a power function (e.g., y = x). n Alternatively, a direct proportional function (e.g., y = kx), or a combination of a power function and a direct proportional function (e.g., y = kx). n Alternatively, a function represented by a neural network can be a function that takes a single pixel value as input and a single pixel value as output, with a neural network as an intermediate layer. It should be noted that the foregoing examples describe several forms of functions, but do not limit the adjustment function. The embodiments of this application do not specifically limit the type and composition of the adjustment function.

[0203] Based on the function to be constructed, the corresponding adjustment parameters of the function can be determined. For example, y = x n The corresponding transformation parameters are n, y = kx and v = kx. nThe corresponding transformation parameters are k and n. In this embodiment, the server can adjust the auxiliary image at the pixel level. That is, all pixel values ​​in the auxiliary image can be adjusted using the same type of adjustment function, but the adjustment parameters corresponding to each pixel value can be different. Alternatively, each pixel value in the auxiliary image can be adjusted using different types of adjustment functions, and the adjustment parameters corresponding to each pixel value can be different. For example, the auxiliary image includes 64 pixels, corresponding to 64 pixel values. These 64 pixel values ​​can correspond to 64 sets of adjustment parameters, meaning the adjustment parameters for each of the 64 pixel values ​​are different. Or, these 64 pixel values ​​can correspond to 2 sets of adjustment parameters, meaning some of the adjustment parameters for the 64 pixel values ​​are the same, while the adjustment parameters for another part of the pixel values ​​are different. Or, these 64 pixel values ​​can correspond to 3 or more sets (less than 64 sets) of adjustment parameters, meaning the 64 pixel values ​​are divided into different sets, the adjustment parameters for pixel values ​​in the same set are the same, while the adjustment parameters for pixel values ​​in different sets are different. In other words, any set of adjustment parameters can correspond to one or more pixel values ​​in the auxiliary image.

[0204] Once a set of adjustment parameters is determined, an adjustment function can be constructed based on this set of parameters. This adjustment function is used to adjust the pixels of the corresponding auxiliary image or auxiliary image block. The corresponding pixel value is input into the adjustment function, and the output of the adjustment function is regarded as the adjusted pixel value of the corresponding pixel. After all pixel values ​​in the auxiliary image have been adjusted according to the aforementioned process, all the adjusted pixel values ​​constitute the adjusted auxiliary image.

[0205] In one embodiment, the server can determine the adjustment parameters of a first pixel (which can be any pixel in the auxiliary image) in the auxiliary image by comparing bitstream sizes. For example, multiple sets of adjustment parameters can be preset, and multiple adjustment functions can be constructed. These multiple adjustment functions are then used to adjust the first pixel, resulting in multiple adjusted pixel values. The residuals between these multiple adjusted pixel values ​​and the corresponding pixel values ​​in the image to be encoded are calculated, yielding multiple residuals. These residuals are then encoded, and the bitstream sizes after encoding are measured. The adjustment parameter corresponding to the smallest bitstream is determined as the adjustment parameter of the first pixel.

[0206] In one embodiment, image boundaries in the rendered result may originate from the geometric boundaries of objects or boundaries created by lighting. On either side of these boundaries, the degree of matching between the auxiliary image and the rendered result may differ. This is because, for example, the relationship between the alpha image and the color of the rendered result is typically a combination of the alpha image, lighting, and material properties (e.g., roughness / smoothness) that constitute the color of the rendered result. Different objects on either side of the boundary may be spatially distant (e.g., object A occludes object B, but the spatial distance between objects A and B is significant), and this spatial distance leads to differences in lighting. Under different lighting conditions, the mapping relationship between the alpha image and the rendered result is different. An alpha image multiplied by 0.1 might be very similar to the rendered result in detail when it's dark, but it might need to be multiplied by 2 when it's bright. Furthermore, the situation is similar when the alpha image and the rendered result have different materials. Therefore, the adaptation function used for pixel values ​​near the boundary should also be different.

[0207] By selecting appropriate adjustment functions and parameters for each pixel value in the auxiliary image, the auxiliary image of the region can be finely adjusted based on the feature matching degree of the region. This results in a higher similarity between the adjusted auxiliary image and the image to be encoded. Consequently, in the subsequent redundancy removal process, more redundant information between the auxiliary image and the image to be encoded can be reduced, thus improving coding performance.

[0208] As described above, the auxiliary images of the image to be encoded may include one or more. Therefore, the server can use the above method to adjust each auxiliary image to obtain one or more adjusted auxiliary images of the image to be encoded.

[0209] In one embodiment, the redundancy removal process employed by the server may include: when there is only one auxiliary image, calculating the residual between the adjusted auxiliary image and the image to be encoded (e.g., the difference in pixel values ​​at corresponding positions), that is, calculating the difference in pixel values ​​at corresponding positions in the adjusted auxiliary image and the image to be encoded, thereby forming a redundancy removal image.

[0210] In another embodiment, when there are multiple auxiliary images, the residual is calculated between one of the adjusted auxiliary images and the image to be encoded. That is, the difference between the pixel values ​​at corresponding positions in the adjusted auxiliary image and the image to be encoded is calculated. Then, the difference between the pixel value in another adjusted auxiliary image and the difference between the pixel values ​​at the corresponding positions is calculated. This process is repeated until all auxiliary images have been calculated. The resulting pixel differences form the deredundant image.

[0211] In another embodiment, the server can first perform a frequency domain transformation on the auxiliary image to obtain an auxiliary image frequency map; then, based on the auxiliary image frequency map, perform redundancy removal processing on the image frequency map to be encoded to obtain a redundancy-removed image frequency map; finally, perform an inverse frequency domain transformation on the redundancy-removed image frequency map to obtain the redundancy-removed image. That is, the server performs adjustments in the frequency domain, instead of applying different adjustment functions to different regions of the same image, but rather applying different adjustment functions to details at different frequencies of the image (performing block-based adjustment in the frequency domain is equivalent to adjusting by frequency level in the original image domain). Although the principle differs, performing adjustments in the frequency domain can still achieve redundancy removal using information from the rendered intermediate images. Moreover, adjusting details by frequency, since any frequency signal will affect the pixel values ​​of the entire image, may result in an image with better spatial continuity, potentially further improving encoding and decoding performance in some scenarios.

[0212] Step 403: Encode the deredundant image to obtain the bitstream.

[0213] The server can encode the deredundant images using a standard video encoder to obtain a bitstream. This standard video encoder can be implemented according to industry video compression standards, such as ITU-T H.264, H.265 / HEVC, H.266 / VVC, AVS2, AVS3, or extensions of such standards. The technology in this application is not limited to any specific encoding standard.

[0214] In one embodiment, the bitstream may include adjustment parameters for adjusting auxiliary images.

[0215] In one embodiment, the bitstream may further include the correspondence between the aforementioned at least one set of adjustment parameters and auxiliary images, auxiliary image blocks, or pixel values ​​in auxiliary images or auxiliary image blocks.

[0216] For example, the server can write adjustment parameter values ​​into the bitstream, and then the terminal device can construct the final adjustment function by inputting the aforementioned adjustment parameter values ​​according to the pre-agreed adjustment function.

[0217] In one embodiment, the bitstream may include a pre-built tuning function.

[0218] In one embodiment, the bitstream may include indexes related to adjustment parameters or adjustment functions. The terminal device can obtain the adjustment parameters or adjustment functions corresponding to the indexes based on a pre-agreed correspondence, thereby obtaining the final adjustment function. It should be noted that embodiments of this application may employ various methods to transmit adjustment parameters or adjustment functions, and no specific limitation is made thereto.

[0219] In one embodiment, the bitstream may further include CGMV. As mentioned above, CGMV is an intermediate rendering result and can therefore be obtained based on rendering parameters. The redundancy removal process in this embodiment still retains spatial motion logic similar to the original image, thus enabling good cooperation with existing CGMV-assisted encoding and decoding algorithms.

[0220] In one embodiment, the bitstream also includes a rendering intermediate result, an auxiliary image, or an adjusted auxiliary image corresponding to the image to be encoded. The server can directly transmit the generated rendering intermediate result to the terminal device without transmitting rendering parameters, allowing the terminal device to directly decode the stream and obtain the rendering intermediate result without rendering processing, thus improving decoding performance. Alternatively, the server can transmit the auxiliary image to the terminal device without transmitting rendering parameters. Or, the server can transmit the adjusted auxiliary image to the terminal device without transmitting adjustment parameters, thereby improving the decoding efficiency of the terminal device.

[0221] Based on the above steps, the server processing procedure in this embodiment of the application can be as follows: Figure 7 ( Figure 7 The following is an exemplary description of the encoding end (as shown in the flowchart of an embodiment of this application). The encoding end (corresponding to the server) encodes several frames, and the compressed image data and adjustment parameters of each frame are written into the bitstream. The encoding process for one frame of image includes the following steps:

[0222] 1. Obtain auxiliary images (corresponding to step 401)

[0223] Examples of embodiments in this application demonstrate that intermediate rendering results can be used as auxiliary images, such as normal maps, albedo maps, etc.

[0224] 2. Calculate the adjustment parameters (corresponding to the step of obtaining the adjustment parameters in step 402).

[0225] The encoding end can determine the adjustment parameters of the first pixel in the auxiliary image (which can be any pixel in the auxiliary image) by comparing the bitstream size. For example, multiple sets of adjustment parameters can be preset, and multiple adjustment functions can be constructed. By applying these multiple adjustment functions to adjust the first pixel, multiple adjusted pixel values ​​of the first pixel can be obtained. The residuals between the multiple adjusted pixel values ​​of the first pixel and the corresponding pixel values ​​in the image to be encoded can be calculated, resulting in multiple residuals. These residuals are encoded, and the bitstream size of the encoded residuals is measured. The adjustment parameter corresponding to the smallest bitstream is determined as the adjustment parameter of the first pixel. The same method can be used to obtain the adjustment parameters of all pixels in the auxiliary image, which are then written into the bitstream and transmitted to the decoding end.

[0226] 3. Adjustment (corresponding to step 402, which is based on obtaining the adjusted auxiliary image)

[0227] The encoding end adjusts the pixel values ​​of the auxiliary image according to the adjustment parameters to better match the image to be encoded. That is, a function (also called an adjustment function) is constructed for each pixel value of the auxiliary image based on the adjustment parameters. Then, the corresponding adjustment function is applied to each pixel in the auxiliary image, with its pixel value as the input of the adjustment function and the output value of the adjustment function as the adjusted value of that pixel, thus forming the adjusted auxiliary image.

[0228] Then, the pixel values ​​at corresponding positions in the adjusted auxiliary image and the image to be encoded are subtracted to obtain the deredundant image, which is then fed into the encoding module.

[0229] 4. Encoding (corresponding to step 403)

[0230] The encoding end receives the deredundant image, encodes it using a standard video encoder to obtain the corresponding bitstream, and then transmits it to the decoding end.

[0231] In this embodiment, the encoding end generates a bitstream and transmits it to the decoding end. An exemplary bitstream structure can be as follows:

[0232] 1) Data Header (Optional): Used to record metadata of the bitstream.

[0233] 2) Block information (optional): Used to enable the decoding end to perform the same image block operation as the encoding end. In one embodiment, there is an operation of dividing the image into blocks by region before further processing; in another embodiment, the image block pattern can be determined by the encoding end, in which case the block information can be transmitted to the decoding end.

[0234] 3) Adjustment parameters (optional): Used to control the adjustment process at the decoding end. The decoding end adjusts the pixel values ​​of the auxiliary image based on the adjustment parameters to restore the image information subtracted at the encoding end.

[0235] 4) Deredundant Image (Required): The deredundant image generated after redundancy removal at the encoding end, and then encoded by the video encoder to produce the resulting bitstream data. The decoding end can use this bitstream to generate the final reconstructed image for display.

[0236] In this embodiment of the application, by using the intermediate rendering result as an auxiliary image or by preprocessing the intermediate rendering result to obtain an auxiliary image, since the auxiliary image has a high similarity to the image to be encoded, the image to be encoded can be deredundantly processed based on the auxiliary image, which can reduce more redundant information in the image to be encoded and improve encoding performance.

[0237] Figure 6This is a flowchart of process 600 of the image decoding method provided in an embodiment of this application. Process 600 can be executed by the terminal device described above. Process 600 is described as a series of steps or operations, and it should be understood that process 600 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 6 The execution order is shown. Process 600 may include:

[0238] Step 601: Receive the bitstream.

[0239] Step 602: Obtain the deredundancy image based on the bitstream.

[0240] The terminal device can decode the bitstream using a standard video decoder to obtain a deredundant image. This standard video decoder can be implemented according to industry video compression standards, such as ITU-T H.264, H.265 / HEVC, H.266 / VVC, AVS2, AVS3, or extensions of such standards. The technology in this application is not limited to any specific decoding standard.

[0241] Step 603: Obtain auxiliary images.

[0242] In one embodiment, the auxiliary image can be obtained based on intermediate rendering results (which may be obtained during the rendering process of the 3D scene by the terminal device, or from the decoded bitstream). The auxiliary image can be taken from the intermediate rendering results or a portion of the intermediate rendering results, or the auxiliary image can be obtained by preprocessing the intermediate rendering results or a portion of the intermediate rendering results.

[0243] In one embodiment, the auxiliary image may also be obtained from the decoded bitstream.

[0244] The deredundant image and auxiliary image can refer to the entire frame of the image or a local image of a frame, such as an image patch or a region of interest (ROI). The size and shape of the local image are not specifically limited.

[0245] Step 604: Obtain the reconstructed image based on the auxiliary image and the deredundancy image.

[0246] The auxiliary images corresponding to the deredundancy image can include one or more.

[0247] In one embodiment, obtaining a reconstructed image based on an auxiliary image and a deduplication image may include: when there is only one auxiliary image, reconstructing based on the auxiliary image and the deduplication image (e.g., summing the pixel values ​​at corresponding positions), that is, calculating the sum of the pixel values ​​at corresponding positions in the auxiliary image and the deduplication image to form a reconstructed image.

[0248] In one embodiment, obtaining a reconstructed image based on an auxiliary image and a deredundancy image may include: when there are multiple auxiliary images, summing one of the auxiliary images and the deredundancy image, that is, calculating the sum of the pixel values ​​at corresponding positions in the auxiliary image and the deredundancy image, and then calculating the sum of the pixel values ​​in another auxiliary image and the sum of the pixel values ​​at the aforementioned corresponding positions, and so on, until all auxiliary images have been calculated, and the final pixel sums form the reconstructed image.

[0249] In one embodiment, the terminal device may first acquire the adjusted auxiliary image, and then reconstruct the image based on the adjusted auxiliary image and the deduplicated image to obtain the reconstructed image.

[0250] In one embodiment, the adjustment method used by the terminal device for the auxiliary image may include: obtaining at least one set of adjustment parameters based on the bitstream; constructing at least one adjustment function based on the at least one set of adjustment parameters; and adjusting the pixels of the auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image. Optionally, one or more pixels in the auxiliary image correspond to a set of adjustment parameters in the at least one set of adjustment parameters, and the corresponding pixels in the auxiliary image are adjusted according to the at least one adjustment function to obtain the adjusted auxiliary image.

[0251] In another embodiment, the adjustment method used by the terminal device for the auxiliary image may include: obtaining a set of adjustment parameters corresponding to an auxiliary image block based on the bitstream, wherein the set of adjustment parameters corresponds to an auxiliary image block or one or more pixels in an auxiliary image block, and the auxiliary image includes multiple auxiliary image blocks; constructing at least one adjustment function based on at least one set of adjustment parameters; adjusting the pixels of the corresponding auxiliary image blocks according to the at least one adjustment function to obtain an adjusted auxiliary image block corresponding to the auxiliary image block; and obtaining an adjusted auxiliary image based on the multiple adjusted auxiliary image blocks. Optionally, a set of adjustment parameters corresponding to an auxiliary image block can be obtained based on the correspondence between auxiliary image blocks and adjustment parameters in the bitstream.

[0252] Optionally, the terminal device can also perform block processing on the auxiliary image. In one embodiment, the block processing method is determined according to the block processing mode, which can be preset or obtained from the bitstream. Specific block processing methods and block modes can be found in the server-side description above.

[0253] Optionally, the bitstream received by the terminal device includes adjustment parameters. The terminal device can construct the final adjustment function based on a pre-agreed adjustment function and the aforementioned adjustment parameter values. Optionally, the bitstream received by the terminal device contains a constructed adjustment function, which the terminal device can obtain by decoding the bitstream. Optionally, the bitstream received by the terminal device contains an index related to the adjustment parameters or adjustment function. The terminal device can obtain the adjustment parameters or adjustment function corresponding to the index based on a pre-agreed correspondence, thereby obtaining the final adjustment function. It should be noted that the embodiments of this application can use various methods to transmit adjustment parameters or adjustment functions, and no specific limitation is made thereto.

[0254] The adjustment function mentioned above can include any one or a combination of the following functions: proportional function, linear function, polynomial function, power function, or function represented by a neural network. For example, a power function (e.g., y = x). n Alternatively, a direct proportional function (e.g., y = kx), or a combination of a power function and a direct proportional function (e.g., y = kx). n Alternatively, a function represented by a neural network can be a function that takes a single pixel value as input and a single pixel value as output, with a neural network as an intermediate layer. It should be noted that the foregoing examples describe several forms of functions, but do not limit the adjustment function. The embodiments of this application do not specifically limit the type and composition of the adjustment function.

[0255] The adjustment parameters corresponding to this adjustment function can be obtained from the bitstream. For example, y = x n The corresponding transformation parameters are n and k, respectively. nThe corresponding transformation parameters are k and n. In this embodiment, the terminal device can adjust the auxiliary image at the pixel level. That is, all pixel values ​​in the auxiliary image can be adjusted using the same type of adjustment function, but the adjustment parameters corresponding to each pixel value can be different. Alternatively, each pixel value in the auxiliary image can be adjusted using different types of adjustment functions, and the adjustment parameters corresponding to each pixel value can be different. For example, the auxiliary image includes 64 pixels, corresponding to 64 pixel values. These 64 pixel values ​​can correspond to 64 sets of adjustment parameters, meaning the adjustment parameters for each of the 64 pixel values ​​are different. Or, these 64 pixel values ​​can correspond to 2 sets of adjustment parameters, meaning some of the adjustment parameters for the 64 pixel values ​​are the same, while the adjustment parameters for another part of the pixel values ​​are different. Or, these 64 pixel values ​​can correspond to 3 or more sets (less than 64 sets) of adjustment parameters, meaning the 64 pixel values ​​are divided into different sets, the adjustment parameters for pixel values ​​in the same set are the same, while the adjustment parameters for pixel values ​​in different sets are different. In other words, any set of adjustment parameters can correspond to one or more pixel values ​​in the auxiliary image. For example, the auxiliary image is divided into multiple auxiliary image blocks. Each auxiliary image block includes 16 pixels, corresponding to 16 pixel values. These 16 pixel values ​​can correspond to 16 sets of adjustment parameters, meaning the adjustment parameters for each of the 16 pixel values ​​are different. Alternatively, these 16 pixel values ​​can correspond to 2 sets of adjustment parameters, meaning some of the adjustment parameters are the same, while others are different. Or, these 16 pixel values ​​can correspond to 3 or more sets (less than 16 sets) of adjustment parameters, meaning the 16 pixel values ​​are divided into different sets, with the same adjustment parameters for pixel values ​​in the same set and different adjustment parameters for pixel values ​​in different sets. For each auxiliary image block, the terminal device can obtain at least one set of adjustment parameters, thereby constructing at least one adjustment function. The adjustment function is then used to adjust the pixel values ​​in the corresponding auxiliary image block to obtain the adjusted auxiliary image block. Finally, multiple adjusted auxiliary image blocks are combined into a complete frame of adjusted auxiliary image.

[0256] After the terminal device obtains a set of adjustment parameters through decoding the bitstream, it can construct an adjustment function based on these parameters. This adjustment function is used to adjust the corresponding pixel values. The corresponding pixel value is input into the adjustment function, and the output of the function is considered the adjusted pixel value. Once all pixel values ​​in the auxiliary image have been adjusted according to the aforementioned process, all the adjusted pixel values ​​constitute the adjusted auxiliary image.

[0257] In one possible implementation, the image boundaries in the rendered result may originate from the geometric boundaries of objects or boundaries created by lighting. On either side of these boundaries, the degree of matching between the auxiliary image and the rendered result may differ. For example, the relationship between the alpha image and the color of the rendered result is typically that the alpha image, lighting, and material properties (e.g., roughness / smoothness) collectively constitute the color of the rendered result. Different objects on either side of the boundary may be spatially distant (e.g., object A occludes object B, but the spatial distance between objects A and B is significant), and this spatial distance leads to differences in lighting. Under different lighting conditions, the mapping relationship between the alpha image and the rendered result is different. An alpha image multiplied by 0.1 might be very similar to the rendered result in detail when it's dark, but it might need to be multiplied by 2 when it's bright. Furthermore, the situation is similar when the materials of the alpha image and the rendered result differ. Therefore, the adaptation function used for pixel values ​​near the boundary will also differ.

[0258] Terminal devices can directly obtain adjustment parameters from the bitstream to construct corresponding adjustment functions. After verification by the server, such adjustment functions can perform detailed adjustments to the auxiliary image of the region based on the feature matching degree of the region. This results in the adjusted auxiliary image having a higher similarity to the image to be encoded. The terminal device can then reproduce the aforementioned adjustment function to achieve auxiliary image adjustment consistent with the server, thereby improving decoding performance.

[0259] After the above processing, the adjusted auxiliary image may include one or more. The reconstruction processing adopted by the terminal device may include: when there is only one adjusted auxiliary image, reconstruction is based on the adjusted auxiliary image and the deduplication image (e.g., summing the pixel values ​​at corresponding positions), that is, calculating the sum of the pixel values ​​at corresponding positions in the adjusted auxiliary image and the deduplication image to form a reconstructed image.

[0260] When there are multiple adjusted auxiliary images, sum the values ​​of one of the adjusted auxiliary images and the deduplication image. That is, calculate the sum of the pixel values ​​at corresponding positions in the adjusted auxiliary image and the deduplication image. Then calculate the sum of the pixel values ​​in another adjusted auxiliary image and the sum of the pixel values ​​at the corresponding positions mentioned above. Continue in this way until all the adjusted auxiliary images have been calculated. The final pixel sums form the reconstructed image.

[0261] In one possible implementation, the bitstream also includes rendering intermediate results or auxiliary images. The terminal device directly decodes the stream to obtain the rendered intermediate results or auxiliary images without performing rendering processing, which can improve decoding performance. Alternatively, the terminal device directly decodes the stream to obtain the adjusted auxiliary images without performing rendering or adjusting the auxiliary images, which can improve the decoding efficiency of the terminal device.

[0262] Based on the above steps, the processing procedure of the terminal device in this embodiment can be as follows: Figure 8 ( Figure 8 The following is an exemplary description of the decoding end (as shown in the flowchart of an embodiment of this application). The decoding end (corresponding to the terminal device) decodes the bitstream and reconstructs several frames. The bitstream includes compressed image data and adjustment parameters for each frame. The decoding process for one frame of image includes the following steps:

[0263] 1. Obtain auxiliary images (corresponding to step 603)

[0264] In this embodiment of the application, the intermediate rendering result can be used as an auxiliary image. The intermediate rendering result can be, for example, a normal map, an albedo map, etc.

[0265] 2. Video Decoding (corresponding to step 602)

[0266] The decoding end receives the bitstream and uses a standard video decoder to decode it to obtain a de-redundant image and adjust the parameters.

[0267] 3. Adjustment (corresponding to step 604)

[0268] The decoder constructs an adjustment function based on the adjustment parameters, and uses the adjustment function to adjust the corresponding pixel values ​​in the auxiliary image to obtain the adjusted auxiliary image, so as to match the adjustment made to the auxiliary image by the encoder.

[0269] Then, the pixel values ​​at corresponding positions in the adjusted auxiliary image and the deredundant image are added together to restore the redundant information removed at the encoding end, thus obtaining the reconstructed image.

[0270] In this embodiment of the application, since the auxiliary image and the image to be encoded have a high similarity in the server, the redundant image obtained by performing redundancy removal processing on the image to be encoded based on the auxiliary image can reduce more redundant information in the image to be encoded. Encoding such a redundant image into a bitstream and transmitting it to the terminal device can achieve the effect of saving bitstream. The terminal device can obtain the reconstructed redundant image through the decoding stream, and reconstruct the reconstructed image by combining it with the auxiliary image, which can also improve decoding performance.

[0271] The following are several specific embodiments to illustrate... Figure 4 and Figure 6 The technical solutions of the illustrated method embodiments will be described in detail.

[0272] Figure 9This is a schematic diagram of the edge-cloud encoding / decoding architecture according to an embodiment of this application. In a graphics rendering engine, the inherent color of each object is often described by albedo. During the calculation and rendering of an image, an albedo map of the rendered object is generated, and further calculations and rendering are performed based on the albedo map. As a description of the inherent color of an object, the albedo map often has a high visual similarity to the final rendering result and contains a large amount of information about the final rendering result. In cloud gaming scenarios, by pre-deploying resources, the graphics rendering engine on the edge can render an albedo map synchronized with the cloud at a lower cost. Therefore, this embodiment utilizes the edge-cloud synchronized albedo map to assist in image encoding and decoding, thereby improving the encoding and decoding performance of the rendered image.

[0273] like Figure 9 As shown, in the edge-cloud encoding / decoding architecture, the redundancy removal module on the cloud side (encoding end) includes... Figure 7 The illustrated embodiment includes three modules: acquiring auxiliary images, calculating adjustment parameters, and adjustment; the end-side (decoding end) reconstruction module includes... Figure 8 The system comprises two modules: acquiring auxiliary images and making adjustments. Rendering parameters include various parameters used for rendering, the position and color vectors of all light sources, the position vector of the player or observer, the sampling method of each texture, the position coordinates of each scene object, the motion trajectory of moving objects, and skeletal animation parameters. Some of these parameters are owned by the client, some by the cloud, and some are shared by both the client and cloud; therefore, rendering parameters need to be synchronized before each frame is rendered. This embodiment does not limit the synchronization method or content of the rendering parameters.

[0274] The cloud-side encoding process includes the following steps:

[0275] 1. Rendering parameter synchronization and acquisition

[0276] a) Transmit rendering parameters

[0277] i. Write the rendering parameters that need to be provided to the client side during the cloud-edge synchronization into the bitstream and transmit it to the client side.

[0278] b) Obtain rendering parameters

[0279] i. Decode the rendering parameter bitstream uploaded from the client and use it as rendering parameters.

[0280] ii. Load the rendering assets pre-stored on disk / in memory as rendering parameters.

[0281] 2. Generate rendering results

[0282] a) Input the rendering parameters obtained in step 1 into the graphics rendering engine (e.g., V-Ray, Unreal, Unity, etc., which have rendering capabilities). The graphics rendering engine executes the rendering process according to the rendering parameters, renders the objects in the scene, and performs shadow and lighting calculations for rendering shading, thereby generating the rendering result as the image to be encoded.

[0283] 3. Generate intermediate rendering results

[0284] In this example, rendering the intermediate result refers to the Albedo plot. Generating the Albedo plot involves the following steps:

[0285] i. Input the rendering parameters obtained in step 1 into the graphics rendering engine (e.g., V-Ray, Unreal, Unity, and other applications with rendering capabilities). The graphics rendering engine executes the rendering process according to the rendering parameters, renders the objects in the scene, and when determining the color of a pixel, uses the Albedo map color value of the point on the object corresponding to that pixel as the color of that pixel, thereby obtaining the Albedo map in the intermediate rendering result.

[0286] 4. Redundancy removal

[0287] a) Acquiring auxiliary images

[0288] Use the Albedo image generated in step 3 as an auxiliary image. Optionally, high-pass filtering can be applied to the Albedo image, including the following steps:

[0289] 1) Calculate the low-frequency representation for each Albedo diagram.

[0290] 2) Subtract the corresponding low-frequency representation from each auxiliary image, and use the result as the final auxiliary image.

[0291] This step removes low-frequency information from the auxiliary image, so that only high-frequency features are adjusted during the redundancy removal process, thus ensuring that the deredundant image remains compatible with existing video coding methods.

[0292] b) Calculate adjustment parameters

[0293] i. Divide the auxiliary image and the image to be encoded into multiple image blocks at a certain resolution (e.g., 128×128).

[0294] ii. Obtain a suitable adjustment function for each auxiliary image patch. In this embodiment, a suitable adjustment function can be a proportional function, such as:

[0295] f(x) = kx

[0296] That is, the pixel value in the auxiliary image block is scaled by a factor of k.

[0297] A suitable adjustment function can ensure a high similarity between the auxiliary image mapped by the adjustment function and the image to be encoded. The similarity can be obtained by calculating the residual between the two images, performing image encoding on the residual, and measuring the size of the encoded bitstream. For example, multiple k values ​​can be enumerated, and the optimal k value can be selected as the optimal adjustment parameter for the auxiliary image.

[0298] iii. The k values ​​corresponding to different pixel values ​​in the auxiliary image constitute multiple sets of adjustment parameters in this embodiment. These multiple sets of adjustment parameters are transmitted to the adjustment module and also written into the bitstream and transmitted to the end side.

[0299] c) Adjustment

[0300] i. For each pixel value in the auxiliary image map, interpolate and filter to obtain the adjustment parameters for that pixel value based on the adjustment parameters of the image block to which the pixel value belongs and the adjustment parameters of the adjacent image blocks stored in the adjustment parameters.

[0301] To conserve bitstream resources, the adjustment parameters for each pixel value in the auxiliary image cannot all be written into the bitstream and sent to the endpoint. As an example, the adjustment parameters for the central pixel value of each auxiliary image or each auxiliary image block within an auxiliary image can be written into the bitstream, while the adjustment parameters for other pixel values ​​in the auxiliary image or its blocks are not written. The endpoint can decode the stream to obtain the adjustment parameters for the central pixel value of each auxiliary image or its blocks, and then use interpolation to obtain the adjustment parameters for the other pixel values ​​in the auxiliary image or its blocks. The reason for using interpolation instead of directly using the adjustment parameters for the central pixel value of the current image block is primarily to prevent abrupt changes at the edges of the image blocks.

[0302] ii. Construct an adjustment function for the pixel value based on the function coefficients of the pixel value.

[0303] iii. After mapping the pixel value through the corresponding adjustment function, the adjusted pixel value is obtained, and finally the adjusted auxiliary image is obtained.

[0304] d) Generate a deredundancy image

[0305] The pixel values ​​at corresponding positions in the adjusted auxiliary image output by the adjustment module and the image to be encoded are subtracted. When there are multiple auxiliary images, the residual is calculated between one of the adjusted auxiliary images and the image to be encoded, that is, the difference between the pixel values ​​at corresponding positions in the adjusted auxiliary image and the image to be encoded is calculated. Then the difference between the pixel value in another adjusted auxiliary image and the difference between the pixel values ​​at the corresponding positions mentioned above is calculated. This process is repeated until all auxiliary images have been calculated. The final pixel differences are used to form the deredundant image.

[0306] 5. Encoding

[0307] a) After the deredundancy processing module generates the deredundancy image, it is encoded using a standard video encoding process to obtain the deredundancy image bitstream.

[0308] 6. Pack the bitstream and transmit it to the terminal. The bitstream includes:

[0309] a) Adjustment parameters used in the adjustment.

[0310] b) The bitstream of the deredundant image generated by video encoding.

[0311] The decoding process on the device side includes the following steps:

[0312] 1. Rendering parameter synchronization and acquisition

[0313] a) Transmit rendering parameters

[0314] i. Write the rendering parameters that need to be provided to the cloud side during the end-to-cloud synchronization into the bitstream and transmit it to the cloud side.

[0315] b) Obtain rendering parameters

[0316] i. Decode the rendering parameter bitstream distributed from the cloud side and use it as rendering parameters.

[0317] ii. Load the rendering assets pre-stored on disk / in memory as rendering parameters.

[0318] 2. Generate intermediate rendering results

[0319] This step is the same as step 3 on the cloud side.

[0320] 3. Decoding the bitstream yields:

[0321] a) Adjustment parameters used in the adjustment.

[0322] b) Deredundant image bitstream.

[0323] 4. Decoding

[0324] a) After obtaining the bitstream of the deredundant image, decode it using a standard video decoder to obtain the reconstructed deredundant image.

[0325] 5. Reconstruction

[0326] a) Acquiring auxiliary images

[0327] This step is consistent with step 4.a) on the cloud side.

[0328] b) Adjustment

[0329] Based on the adjustment parameters obtained from the decoded bitstream, perform the same adjustments as step 4.c) on the cloud side to obtain the adjusted auxiliary image.

[0330] c) Generate reconstructed image

[0331] The pixel values ​​at corresponding positions in the decoded deredundancy image and the adjusted auxiliary image are added together. When there are multiple auxiliary images, the adjusted auxiliary image and the deredundancy image are summed, that is, the sum of the pixel values ​​at corresponding positions in the adjusted auxiliary image and the deredundancy image is calculated. Then the pixel values ​​in the other adjusted auxiliary image are summed with the sum of the pixel values ​​at the corresponding positions mentioned above. This process is repeated until all auxiliary images have been calculated. The final pixel sums form the reconstructed image.

[0332] This embodiment utilizes an Albedo image generated by edge-cloud collaborative rendering as an auxiliary image. Only adjustment parameters need to be transmitted, reducing the amount of video information directly transmitted in scenarios such as cloud gaming. Therefore, compared to directly encoding video, bitstream consumption is reduced. The pixel values ​​of the auxiliary image are adjusted, where the adjustment parameters used in the adjustment function are calculated from both the auxiliary image and the image to be encoded. These adjustment parameters effectively encompass redundant information from the image to be encoded, and this information is transmitted to the decoding end in the form of an adjustment parameter bitstream. Therefore, the encoding and decoding performance is improved when the Albedo image is used to assist in video encoding and decoding. For example, Figure 10 This is a comparison chart of the rendering results, Albedo diagram, and redundancy-free rendering results in this embodiment.

[0333] In another embodiment, a normal diagram can be used instead. Figure 9 The Albedo image in the illustrated embodiment serves as an auxiliary image. In this embodiment, both the edge and cloud sides possess normal maps as rendering assets; rendering parameters include the scene's 3D model, lighting information, normal maps, and camera parameters, and the only rendering parameter that needs to be synchronized is still the user's camera parameters.

[0334] Related to the above Figure 9 The difference in the illustrated embodiment is that this embodiment uses the normal map in the generated intermediate rendering result as an auxiliary image.

[0335] Optionally, in one embodiment, during the step of acquiring the auxiliary image on the cloud side, a high-pass filter can be applied to the normal map to obtain the final auxiliary image. The specific high-pass filtering includes the following steps:

[0336] 1) Calculate the low-frequency representation for each normal plot.

[0337] 2) Subtract the corresponding low-frequency representation from each auxiliary image, and use the result as the final auxiliary image.

[0338] This step removes low-frequency information from the auxiliary image, so that only high-frequency features are adjusted during the redundancy removal process, thus ensuring that the deredundant image remains compatible with existing video coding methods.

[0339] Apart from the differences mentioned above, the other steps in this embodiment can be referred to. Figure 9 The relevant steps in the illustrated embodiments will not be repeated here.

[0340] This embodiment utilizes a normal map generated by edge-cloud collaborative rendering as an auxiliary image. Only adjustment parameters need to be transmitted, reducing the amount of video information directly transmitted in scenarios such as cloud gaming. Therefore, compared to directly encoding video, bitrate consumption is reduced. The pixel values ​​of the auxiliary image are adjusted, and the adjustment parameters used in the adjustment function are calculated from both the auxiliary image and the image to be encoded. These adjustment parameters effectively encompass redundant information from the image to be encoded, and this information is transmitted to the decoding end in the form of an adjustment parameter bitrate. Therefore, the encoding and decoding performance is improved when using the Albedo map to assist in video encoding and decoding. Compared to using only the Albedo map, it is better able to handle image features caused by lighting.

[0341] Figure 11 This is a schematic diagram of the end-to-cloud encoding / decoding architecture according to an embodiment of this application. Figure 9 Based on the illustrated embodiment, CGMV-assisted video encoding and decoding technologies are used to input CGMV information into the encoding and decoding module, thereby using existing CGMV-assisted video encoding and decoding methods to transmit the deredundant image output by this embodiment.

[0342] Related to the above Figure 9 The difference in the illustrated embodiment is that, in this embodiment, the graphics rendering engine additionally generates a CGMV and transmits it to the encoding module. Therefore, in the cloud-side encoding step, the CGMV-assisted video encoding algorithm in related technologies is used to encode the deredundant image and obtain the bitstream of the deredundant image. In the end-side decoding step, after obtaining the bitstream of the deredundant image transmitted from the cloud side and the CGMV generated by the graphics rendering engine, the CGMV-assisted video decoding algorithm in related technologies is used to decode the reconstructed deredundant image.

[0343] Apart from the differences mentioned above, the other steps in this embodiment can be referred to. Figure 9 The relevant steps in the illustrated embodiments will not be repeated here.

[0344] This embodiment utilizes Albedo graphs and CGMVs generated through edge-cloud collaborative rendering, reducing the amount of video information transmitted directly in scenarios such as cloud gaming. Therefore, compared to directly encoding video, bitrate consumption is reduced. Since the deredundant image still retains spatial motion logic similar to the original image, it can work well with existing CGMV-assisted encoding and decoding algorithms.

[0345] Figure 12 This is a schematic diagram of the end-to-cloud encoding / decoding architecture according to an embodiment of this application. Figure 9 The difference in the illustrated embodiment is that rendering is not enabled on the edge side. Instead, the intermediate rendering results generated by the cloud-side graphics rendering engine are encoded by a standard video encoder or other encoder, and then written into the bitstream for transmission to the edge side. Therefore, the bitstream also includes the bitstream of the intermediate rendering results, and the edge side can obtain the intermediate rendering results through the decoder stream.

[0346] Apart from the differences mentioned above, the other steps in this embodiment can be referred to. Figure 9 The relevant steps in the illustrated embodiments will not be repeated here.

[0347] This embodiment splits the rendered image into an intermediate rendering result and a de-redundant image after removing details from the intermediate rendering result, and encodes and transmits both to the end side. In situations where both the intermediate rendering result and the rendered result need to be sent to the end side, it can reduce bandwidth.

[0348] Figure 13 This is a schematic diagram of the edge-cloud encoding / decoding architecture according to an embodiment of this application. In the field of image processing, any image has its corresponding frequency domain representation, and the frequency domain representation of an image can still be represented by an image of the same resolution (hereinafter referred to as a frequency map). The frequency maps of the image to be encoded and the auxiliary image still have similar image features, and the image features of the image to be encoded can be removed in the frequency domain using this embodiment.

[0349] Based on this, the server can perform frequency domain transformation on the auxiliary image and the image to be encoded to obtain the frequency map of the auxiliary image and the image frequency map of the image to be encoded; then, based on the frequency map of the auxiliary image, perform redundancy removal processing on the image frequency map of the image to be encoded to obtain the redundancy-removed image frequency map; and then perform inverse frequency domain transformation on the redundancy-removed image frequency map to obtain the redundancy-removed image.

[0350] In other words, the server performs adjustments in the frequency domain, instead of applying different adjustment functions to different regions of the same image. Instead, it applies different adjustment functions to the details of different frequencies within the image (performing block-by-block adjustments in the frequency domain is equivalent to adjusting the original image domain (i.e., the image domain before frequency domain adjustments) according to frequency levels). Although the principle differs, performing adjustments to the auxiliary image in the frequency domain can still achieve redundancy removal by utilizing information from the intermediate rendering images. Moreover, adjusting frequency details, since any frequency signal affects the pixel values ​​of the entire image, may result in an image with better spatial continuity, potentially further improving encoding and decoding performance in certain scenarios.

[0351] The terminal device can first perform frequency domain transformation on the auxiliary image to obtain the frequency map of the auxiliary image; then perform frequency domain transformation on the deredundant image to obtain the frequency map of the deredundant image; then reconstruct the frequency map of the deredundant image based on the frequency map of the auxiliary image to obtain the frequency map of the reconstructed image; and finally perform inverse frequency domain transformation on the frequency map of the reconstructed image to obtain the reconstructed image.

[0352] In other words, the terminal device performs adjustments in the frequency domain, rather than applying different adjustment functions to different regions of the same image. Instead, it applies different adjustment functions to details at different frequencies of the image (performing block-based adjustments in the frequency domain is equivalent to adjusting by frequency level in the original image domain). Although the principle is different, performing adjustments in the frequency domain can still achieve redundancy removal by utilizing information from the intermediate rendered images. Moreover, adjusting details to be encoded by frequency, since any frequency signal affects the pixel values ​​of the entire image, may result in an image with better spatial continuity, potentially further improving encoding and decoding performance in certain scenarios.

[0353] Related to the above Figure 9 The difference in the illustrated embodiment lies in the addition of frequency domain transformation and inverse transformation steps. On the cloud side, the auxiliary image and the image to be encoded are converted into frequency maps through frequency domain transformation operations (e.g., discrete cosine transform, discrete sine transform, etc.) to obtain the frequency map of the deredundant image. Then, it is inversely transformed back to the original image domain. On the edge side, the auxiliary image and the deredundant image are converted into frequency maps through frequency domain transformation operations (e.g., discrete cosine transform, discrete sine transform, etc.) to obtain the frequency map of the reconstructed image. Then, it is inversely transformed back to the original image domain to obtain the final reconstructed image.

[0354] Apart from the differences mentioned above, the other steps in this embodiment can be referred to. Figure 9 The relevant steps in the illustrated embodiments will not be repeated here.

[0355] This embodiment performs the adjustment operation in the frequency domain, instead of applying different fitness functions to different regions of the same image. Instead, it applies different fitness functions to details at different frequencies of the image (performing block-based adjustment in the frequency domain is equivalent to adjusting by frequency level in the original image domain). Despite the difference in principle, this embodiment can still achieve redundancy removal using information from the rendered intermediate image. Figure 9 Compared to adjusting details by region in the illustrated embodiment, this embodiment adjusts details by frequency. Since any frequency signal will affect the pixel value of the entire image, the image generated by this embodiment may have better spatial continuity and may further improve encoding and decoding performance in some scenarios.

[0356] Figure 14This is a schematic diagram of the structure of the image encoding device 1400 according to an embodiment of this application, as shown below. Figure 14 As shown, the image encoding device 1400 of this embodiment can be applied to the above-mentioned server. The image encoding device 1400 may include a redundancy removal module 1401 and an encoding module 1402. Wherein,

[0357] The redundancy removal module 1401 is used to acquire an auxiliary image, which is obtained based on the rendering intermediate result corresponding to the image to be encoded; the image to be encoded is subjected to redundancy removal processing according to the auxiliary image to obtain a redundancy removal image; the encoding module 1402 is used to encode the redundancy removal image to obtain a bitstream.

[0358] In one embodiment, the redundancy removal module 1401 is specifically used to obtain an adjusted auxiliary image, which is closer to the image to be encoded than the auxiliary image; and to perform redundancy removal processing based on the adjusted auxiliary image and the image to be encoded to obtain the redundancy removal image.

[0359] In one embodiment, the redundancy removal module 1401 is specifically configured to obtain at least one set of adjustment parameters based on the auxiliary image and the image to be encoded; construct at least one adjustment function based on the at least one set of adjustment parameters; and adjust the pixels of the corresponding auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image.

[0360] In one embodiment, the redundancy removal module 1401 is specifically used to divide the auxiliary image into blocks to obtain multiple auxiliary image blocks; for any auxiliary image block, obtain at least one set of adjustment parameters according to the auxiliary image block and the corresponding block in the image to be encoded; construct at least one adjustment function according to the at least one set of adjustment parameters; and adjust the pixels of the corresponding auxiliary image block according to the at least one adjustment function to obtain the adjusted auxiliary image block corresponding to the auxiliary image block.

[0361] In one embodiment, the bitstream further includes the at least one set of adjustment parameters.

[0362] In one embodiment, the adjustment function includes any one or a combination of the following functions: a proportional function, a linear function, a polynomial function, a power function, or a function represented by a neural network.

[0363] In one embodiment, the intermediate rendering result includes at least one of an albedo map, a normal map, a low-quality rendering map, or a position map.

[0364] In one embodiment, the bitstream may further include intermediate rendering results corresponding to the image to be encoded or the auxiliary image.

[0365] In one embodiment, the bitstream further includes block information, which includes the correspondence between the at least one set of adjustment parameters and the plurality of auxiliary image blocks.

[0366] In one embodiment, the redundancy removal module 1401 is further configured to perform frequency domain transformation on the auxiliary image to obtain an auxiliary image frequency map; perform frequency domain transformation on the image to be encoded to obtain an image frequency map to be encoded; perform redundancy removal processing on the image frequency map to be encoded according to the auxiliary image frequency map to obtain a redundancy-removed image frequency map; and perform inverse frequency domain transformation on the redundancy-removed image frequency map to obtain the redundancy-removed image.

[0367] The apparatus of this embodiment can be used to perform Figure 4 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0368] Figure 15 This is a schematic diagram of the structure of the image encoding device 1500 according to an embodiment of this application, as shown below. Figure 15 As shown, the image encoding device 1500 of this embodiment can be applied to the aforementioned terminal device. The image encoding device 1500 may include: a receiving module 1501, a decoding module 1502, and a reconstruction module 1503. Wherein,

[0369] The receiving module 1501 is used to receive the bitstream; the decoding module 1502 is used to obtain a deredundant image based on the bitstream; the reconstruction module 1503 is used to obtain an auxiliary image; and to obtain a reconstructed image based on the auxiliary image and the deredundant image.

[0370] In one embodiment, the reconstruction module 1503 is specifically used to acquire an adjusted auxiliary image, which is closer to the reconstructed image than the auxiliary image; and to reconstruct the reconstructed image based on the adjusted auxiliary image and the deduplicated image.

[0371] In one embodiment, the bitstream further includes at least one set of adjustment parameters; the reconstruction module 1503 is specifically configured to obtain at least one set of adjustment parameters based on the bitstream; construct at least one adjustment function based on the at least one set of adjustment parameters; and adjust the pixels of the corresponding auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image.

[0372] In one embodiment, the bitstream further includes at least one set of adjustment parameters; the reconstruction module 1503 is specifically configured to obtain at least one set of adjustment parameters according to the bitstream, wherein any set of adjustment parameters corresponds to an auxiliary image block, and the auxiliary image includes multiple auxiliary image blocks; construct at least one adjustment function according to the at least one set of adjustment parameters; adjust the pixels of the corresponding auxiliary image blocks according to the at least one adjustment function to obtain an adjusted auxiliary image block corresponding to the auxiliary image block; and obtain the adjusted auxiliary image based on the multiple adjusted auxiliary image blocks.

[0373] In one embodiment, the adjustment function includes any one or a combination of the following functions: a proportional function, a linear function, a polynomial function, a power function, or a function represented by a neural network.

[0374] In one embodiment, the bitstream further includes the intermediate rendering result corresponding to the deredundancy image or the auxiliary image.

[0375] In one embodiment, the auxiliary image is obtained based on the rendering intermediate result corresponding to the deredundancy image; or, the auxiliary image is obtained based on the bitstream.

[0376] In one embodiment, the intermediate rendering result includes at least one of an albedo map, a normal map, a low-quality rendering map, or a position map.

[0377] In one embodiment, the bitstream further includes block information, which includes the correspondence between the at least one set of adjustment parameters and the plurality of auxiliary image blocks.

[0378] In one embodiment, the reconstruction module 1503 is further configured to perform frequency domain transformation on the auxiliary image to obtain an auxiliary image frequency map; perform frequency domain transformation on the deredundant image to obtain a deredundant image frequency map; reconstruct the deredundant image frequency map based on the auxiliary image frequency map to obtain a reconstructed image frequency map; and perform inverse frequency domain transformation on the reconstructed image frequency map to obtain the reconstructed image.

[0379] The apparatus of this embodiment can be used to perform Figure 6 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0380] In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware encoding processor, or implemented by a combination of hardware and software modules in the encoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0381] The memory mentioned in the above embodiments can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0382] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0383] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0384] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0385] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0386] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0387] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0388] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image encoding method, characterized in that, include: Obtain an auxiliary image, which is obtained based on intermediate rendering results obtained during the rendering of the 3D scene, wherein the 3D scene is rendered to obtain the image to be encoded; The image to be encoded is subjected to redundancy removal processing based on the auxiliary image to obtain a redundancy-removed image; The deredundant image is encoded to obtain a bitstream.

2. The method according to claim 1, characterized in that, The step of performing redundancy removal processing on the image to be encoded based on the auxiliary image to obtain a redundancy-removed image includes: Obtain an adjusted auxiliary image, which is closer to the image to be encoded than the original auxiliary image; The adjusted auxiliary image and the image to be encoded are used to perform redundancy removal processing to obtain the redundancy-removed image.

3. The method according to claim 2, characterized in that, The process of acquiring the adjusted auxiliary image includes: At least one set of adjustment parameters is obtained based on the auxiliary image and the image to be encoded; Construct at least one adjustment function based on the at least one set of adjustment parameters; The pixels of the corresponding auxiliary image are adjusted according to the at least one adjustment function to obtain the adjusted auxiliary image.

4. The method according to claim 2, characterized in that, The process of acquiring the adjusted auxiliary image includes: The auxiliary image is divided into blocks to obtain multiple auxiliary image blocks; For any auxiliary image block, at least one set of adjustment parameters is obtained based on the auxiliary image block and the corresponding block in the image to be encoded; Construct at least one adjustment function based on the at least one set of adjustment parameters; The pixels of the corresponding auxiliary image blocks are adjusted according to the at least one adjustment function to obtain the adjusted auxiliary image blocks corresponding to the auxiliary image blocks.

5. The method according to claim 3 or 4, characterized in that, The bitstream also includes the at least one set of adjustment parameters.

6. The method according to any one of claims 3-5, characterized in that, The adjustment function includes any one or a combination of the following functions: a proportional function, a linear function, a polynomial function, a power function, or a function represented by a neural network.

7. The method according to any one of claims 1-6, characterized in that, The intermediate rendering results include at least one of the following: albedo map, normal map, low-quality rendering map, or position map.

8. The method according to any one of claims 1-7, characterized in that, The bitstream also includes the intermediate rendering results corresponding to the image to be encoded or the auxiliary image.

9. The method according to claim 4, characterized in that, The bitstream also includes block information, which includes the correspondence between the at least one set of adjustment parameters and the plurality of auxiliary image blocks.

10. The method according to any one of claims 1-9, characterized in that, Before performing redundancy removal processing on the image to be encoded based on the auxiliary image to obtain a redundancy-free image, the method further includes: The auxiliary image is subjected to frequency domain transformation to obtain an auxiliary image frequency map; The image to be encoded is subjected to frequency domain transformation to obtain the frequency map of the image to be encoded; The step of performing redundancy removal processing on the image to be encoded based on the auxiliary image to obtain a redundancy-removed image includes: The frequency map of the image to be encoded is subjected to redundancy removal processing based on the auxiliary image frequency map to obtain a redundancy-removed image frequency map. The frequency domain inverse transform of the frequency map of the deredundant image is performed to obtain the deredundant image.

11. An image decoding method, characterized in that, include: Receive the bitstream; Obtain the deredundant image based on the bitstream; Acquire auxiliary images, which are obtained based on intermediate rendering results obtained during the rendering of the 3D scene; The reconstructed image is obtained based on the auxiliary image and the deredundancy-reduced image.

12. The method according to claim 11, characterized in that, The step of obtaining the reconstructed image based on the auxiliary image and the deredundancy-reduced image includes: Obtain an adjusted auxiliary image, which is closer to the reconstructed image than the original auxiliary image; The reconstructed image is obtained by reconstructing the image based on the adjusted auxiliary image and the deredundant image.

13. The method according to claim 12, characterized in that, The bitstream also includes at least one set of adjustment parameters; The process of acquiring the adjusted auxiliary image includes: At least one set of adjustment parameters is obtained based on the bitstream; Construct at least one adjustment function based on the at least one set of adjustment parameters; The pixels of the corresponding auxiliary image are adjusted according to the at least one adjustment function to obtain the adjusted auxiliary image.

14. The method according to claim 12, characterized in that, The bitstream also includes at least one set of adjustment parameters; The process of acquiring the adjusted auxiliary image includes: At least one set of adjustment parameters is obtained based on the bitstream, and any set of adjustment parameters corresponds to an auxiliary image block. The auxiliary image includes multiple auxiliary image blocks. Construct at least one adjustment function based on the at least one set of adjustment parameters; The pixels of the corresponding auxiliary image blocks are adjusted according to the at least one adjustment function to obtain the adjusted auxiliary image blocks corresponding to the auxiliary image blocks; The adjusted auxiliary image is obtained from multiple adjusted auxiliary image blocks.

15. The method according to claim 13 or 14, characterized in that, The adjustment function includes any one or a combination of the following functions: a proportional function, a linear function, a polynomial function, a power function, or a function represented by a neural network.

16. The method according to any one of claims 11-15, characterized in that, The bitstream also includes the intermediate rendering result corresponding to the deredundancy image or the auxiliary image.

17. The method according to claim 16, characterized in that, The auxiliary image is obtained based on intermediate rendering results; or, the auxiliary image is obtained based on the bitstream.

18. The method according to claim 16 or 17, characterized in that, The intermediate rendering results include at least one of the following: albedo map, normal map, low-quality rendering map, or position map.

19. The method according to claim 14, characterized in that, The bitstream also includes block information, which includes the correspondence between the at least one set of adjustment parameters and the plurality of auxiliary image blocks.

20. The method according to any one of claims 11-19, characterized in that, Before obtaining the reconstructed image based on the auxiliary image and the deredundancy-free image, the method further includes: The auxiliary image is subjected to frequency domain transformation to obtain an auxiliary image frequency map; The deredundant image is subjected to frequency domain transformation to obtain the deredundant image frequency map; The step of obtaining the reconstructed image based on the auxiliary image and the deredundancy-reduced image includes: The deredundant image frequency map is reconstructed based on the auxiliary image frequency map to obtain the reconstructed image frequency map; The reconstructed image is obtained by performing an inverse frequency domain transform on the frequency map of the reconstructed image.

21. An image encoding device, characterized in that, include: A redundancy removal module is used to acquire auxiliary images, which are obtained based on intermediate rendering results obtained during the rendering process of the 3D scene, and the 3D scene is rendered to obtain the image to be encoded. The image to be encoded is subjected to redundancy removal processing based on the auxiliary image to obtain a redundancy-removed image; The encoding module is used to encode the deredundant image to obtain a bitstream.

22. The apparatus according to claim 21, characterized in that, The redundancy removal module is specifically used to acquire an adjusted auxiliary image, which is closer to the image to be encoded than the auxiliary image; and to perform redundancy removal processing on the adjusted auxiliary image and the image to be encoded to obtain the redundancy removal image.

23. The apparatus according to claim 22, characterized in that, The redundancy removal module is specifically used to obtain at least one set of adjustment parameters based on the auxiliary image and the image to be encoded; construct at least one adjustment function based on the at least one set of adjustment parameters; and adjust the pixels of the corresponding auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image.

24. The apparatus according to claim 22, characterized in that, The redundancy removal module is specifically used to divide the auxiliary image into blocks to obtain multiple auxiliary image blocks; for any auxiliary image block, obtain at least one set of adjustment parameters according to the auxiliary image block and the corresponding block in the image to be encoded; construct at least one adjustment function according to the at least one set of adjustment parameters; and adjust the pixels of the corresponding auxiliary image block according to the at least one adjustment function to obtain the adjusted auxiliary image block corresponding to the auxiliary image block.

25. The apparatus according to claim 23 or 24, characterized in that, The bitstream also includes the at least one set of adjustment parameters.

26. The apparatus according to any one of claims 23-25, characterized in that, The adjustment function includes any one or a combination of the following functions: a proportional function, a linear function, a polynomial function, a power function, or a function represented by a neural network.

27. The apparatus according to any one of claims 21-26, characterized in that, The intermediate rendering results include at least one of the following: albedo map, normal map, low-quality rendering map, or position map.

28. The apparatus according to any one of claims 21-27, characterized in that, The bitstream also includes the intermediate rendering results corresponding to the image to be encoded or the auxiliary image.

29. The apparatus according to claim 24, characterized in that, The bitstream also includes block information, which includes the correspondence between the at least one set of adjustment parameters and the plurality of auxiliary image blocks.

30. The apparatus according to any one of claims 21-29, characterized in that, The redundancy removal module is further configured to perform frequency domain transformation on the auxiliary image to obtain an auxiliary image frequency map; perform frequency domain transformation on the image to be encoded to obtain an image frequency map to be encoded; and perform redundancy removal processing on the image frequency map to be encoded based on the auxiliary image frequency map to obtain a redundancy-removed image frequency map. The frequency domain inverse transform of the frequency map of the deredundant image is performed to obtain the deredundant image.

31. An image decoding device, characterized in that, include: The receiving module is used to receive the bit stream; The decoding module is used to obtain the deduplication image based on the bitstream; The reconstruction module is used to acquire an auxiliary image, which is obtained based on intermediate rendering results obtained during the rendering of the 3D scene; and to acquire a reconstructed image based on the auxiliary image and the deredundancy image.

32. The apparatus according to claim 31, characterized in that, The reconstruction module is specifically used to acquire an adjusted auxiliary image, which is closer to the reconstructed image than the auxiliary image; and to reconstruct the reconstructed image based on the adjusted auxiliary image and the deduplicated image.

33. The apparatus according to claim 32, characterized in that, The bitstream further includes at least one set of adjustment parameters; the reconstruction module is specifically used to obtain at least one set of adjustment parameters based on the bitstream; construct at least one adjustment function based on the at least one set of adjustment parameters; and adjust the pixels of the corresponding auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image.

34. The apparatus according to claim 32, characterized in that, The bitstream further includes at least one set of adjustment parameters; the reconstruction module is specifically used to obtain at least one set of adjustment parameters according to the bitstream, wherein any set of adjustment parameters corresponds to an auxiliary image block, and the auxiliary image includes multiple auxiliary image blocks; construct at least one adjustment function according to the at least one set of adjustment parameters; adjust the pixels of the corresponding auxiliary image blocks according to the at least one adjustment function to obtain an adjusted auxiliary image block corresponding to the auxiliary image block; and obtain the adjusted auxiliary image based on the multiple adjusted auxiliary image blocks.

35. The apparatus according to claim 33 or 34, characterized in that, The adjustment function includes any one or a combination of the following functions: a proportional function, a linear function, a polynomial function, a power function, or a function represented by a neural network.

36. The apparatus according to any one of claims 31-35, characterized in that, The bitstream also includes the intermediate rendering result corresponding to the deredundancy image or the auxiliary image.

37. The apparatus according to claim 36, characterized in that, The auxiliary image is obtained based on the intermediate rendering result corresponding to the deredundancy image; or, the auxiliary image is obtained based on the bitstream.

38. The apparatus according to claim 36 or 37, characterized in that, The intermediate rendering results include at least one of the following: albedo map, normal map, low-quality rendering map, or position map.

39. The apparatus according to claim 34, characterized in that, The bitstream also includes block information, which includes the correspondence between the at least one set of adjustment parameters and the plurality of auxiliary image blocks.

40. The apparatus according to any one of claims 31-39, characterized in that, The reconstruction module is further configured to perform frequency domain transformation on the auxiliary image to obtain an auxiliary image frequency map; perform frequency domain transformation on the deredundant image to obtain a deredundant image frequency map; reconstruct the deredundant image frequency map based on the auxiliary image frequency map to obtain a reconstructed image frequency map; and perform inverse frequency domain transformation on the reconstructed image frequency map to obtain the reconstructed image.

41. An edge-cloud collaborative system, characterized in that, include: Servers and terminal devices; among which... The server is used to render a 3D scene to obtain an image to be encoded and intermediate rendering results generated during the rendering process, and to obtain a first auxiliary image based on the intermediate rendering results; to perform redundancy removal processing on the image to be encoded based on the first auxiliary image to obtain a deredundancy image; to encode the deredundancy image and write the encoded data of the deredundancy image into a bitstream; and to send the bitstream to the terminal device. The terminal device is configured to receive the bitstream; decode the bitstream to obtain a decoding result, the decoding result including a deredundant image; acquire a second auxiliary image, the second auxiliary image being obtained based on an intermediate rendering result of the three-dimensional scene; and reconstruct based on the deredundant image and the second auxiliary image to obtain a reconstructed image.

42. The system according to claim 41, characterized in that, The resolution of the first auxiliary image is different from that of the second auxiliary image; and / or, the content of the first auxiliary image is different from that of the second auxiliary image.

43. The system according to claim 41 or 42, characterized in that, The resolution of the second auxiliary image is higher than that of the first auxiliary image.

44. A device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-20.

45. A computer-readable storage medium, characterized in that, Includes a computer program, which, when executed on the device, causes the device to perform the method of any one of claims 1-20.

46. ​​A computer program product, characterized in that, The computer program product includes computer program code that, when run on a device, causes the device to perform the method of any one of claims 1-20.

47. A bitstream, characterized in that, The bitstream is generated by the method according to any one of claims 1-10.

48. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a bitstream obtained by one or more processors performing the method as described in any one of claims 1-10.

Images