End-cloud coordination system, coding and decoding method, and electronic device

Abstract

Embodiments of the present application provide an end-cloud collaborative system, a coding and decoding method and an electronic device. The coding method comprises: rendering a three-dimensional scene based on rendering parameters to obtain a rendered image; selecting a first rendering intermediate result based on rendering intermediate results generated in the rendering process; encoding the rendered image based on the first rendering intermediate result to obtain coding data of a residual block and writing the coding data of the residual block into a bitstream; the first rendering intermediate result is used for at least one processing in the coding process: blocking, prediction or filtering; the bitstream does not include coding data of the first rendering intermediate result, and a second rendering intermediate result is part of the first rendering intermediate result. In this way, the server can not send the rendering intermediate result to the terminal device, and the interaction delay can be reduced while ensuring that the data stream transmitted by the server to the terminal device is effectively reduced.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of coding and decoding, and in particular to an end-cloud collaborative system, a coding and decoding method, and an electronic device. BACKGROUND

[0002] Many scenarios (such as games, virtual reality (VR) / augmented reality (AR), etc.) need to use rendering to generate pictures, so that the resulting pictures are more realistic and improve the user experience. Among them, rendering requires relatively strong computing power, and the end-side device is limited by objective physical conditions such as device size and power consumption, and the computing power is much weaker than that of the cloud-side server; therefore, rendering is usually deployed on the cloud-side server, the cloud-side server performs rendering, and then the rendered image / video is compressed and sent to the end-side device for display.

[0003] With the increasing demand for rendering quality and the increasing clarity of display devices, the image quality and resolution of the rendered image / video are also constantly improving, which will result in an increase in the code rate overhead of the compressed rendered image / video, and further increase the network bandwidth occupation, resulting in a large interactive delay. In the prior art, the cloud-side server usually encodes and transmits low-resolution images / videos obtained by rendering, and transmits the rendering intermediate results generated in the process of transmitting high-resolution images / videos to the end-side device. The end-side device uses the rendering intermediate results issued by the cloud-side server to upsample the low-resolution images / videos obtained by rendering issued by the cloud-side server, to generate high-resolution images / videos to be displayed and display. In this way, although the code rate overhead can be reduced to some extent, the coding efficiency is still relatively low. SUMMARY

[0004] Therefore, the present application provides an end-cloud collaborative system, a coding and decoding method, and an electronic device. The coding and decoding method is implemented based on the end-cloud collaborative system, which can reduce the interactive delay while ensuring that the code rate overhead of the data stream transmitted from the server to the terminal device is effectively reduced.

[0005] In a first aspect, an embodiment of the present application provides an end-cloud collaborative system, which includes a server and a terminal device. The server includes a first rendering module, an encoder, and a first communication module. The terminal device includes a second communication module, a second rendering module, and a decoder. Wherein:

[0006] The first rendering module is configured to perform rendering processing on a three-dimensional scene based on rendering parameters to obtain a rendered image. The rendering parameters include a first rendering parameter obtained from the terminal device. The first rendering intermediate result is selected based on a rendering intermediate result generated in the rendering processing.

[0007] The encoder is configured to encode the rendered image based on the first rendering intermediate result to obtain encoded data of the residual block and write the encoded data of the residual block into the bitstream; wherein the residual block is obtained by performing residual calculation on the to-be-encoded block and the corresponding prediction block in the rendered image, and the prediction block is obtained by performing prediction on the to-be-encoded block; the first rendering intermediate result is used for at least one of the following processes in the encoding process: blocking, prediction or filtering; and the bitstream does not include encoded data of the first rendering intermediate result;

[0008] The first communication module is configured to send the bitstream;

[0009] The second communication module is configured to receive the bitstream;

[0010] The decoder is configured to parse the bitstream to obtain a parsing result, the parsing result including a residual block corresponding to the current frame;

[0011] The second rendering module is configured to perform rendering processing on the three-dimensional scene based on rendering parameters corresponding to the current frame, and generate the first rendering intermediate result in the rendering processing; wherein the rendering parameters corresponding to the current frame include the first rendering parameter generated by the terminal device;

[0012] The decoder is further configured to reconstruct, based on the first rendering intermediate result generated by the second rendering module and the residual block corresponding to the current frame, a reconstructed image of the current frame; wherein the first rendering intermediate result generated by the second rendering module is used for at least one of the following processes in the reconstruction process: prediction or filtering.

[0013] In this way, all the rendering is performed by the terminal device, and thus the server can not send the rendering intermediate result to the terminal device; therefore, the application can reduce the interactive delay while effectively reducing the code rate overhead of the data stream transmitted by the server to the terminal device. In addition, since the rendering intermediate result has a strong correlation with the rendered image, the application encodes the rendered image based on the rendering intermediate result, thereby ensuring the image reconstruction quality.

[0014] In addition, the encoder involved in the application is a modified (or optimized) encoder of an existing encoder, which can encode the rendered image based on the first rendering intermediate result. In other words, the encoder involved in the application can encode the rendered image based on the first rendering intermediate result, and includes all or part of the functions of the existing encoder. The decoder involved in the application is a modified (or optimized) decoder of an existing decoder, which can decode based on the first rendering intermediate result. In other words, the decoder involved in the application can decode based on the first rendering intermediate result, and includes all or part of the functions of the existing decoder. In this way, the rendering intermediate result can be fully utilized, and the encoding efficiency can be further improved.

[0015] Exemplarily, the server can be a game server, and the server can be a single server or a server cluster, and the application does not limit this.

[0016] Exemplarily, the terminal device includes but is not limited to a personal computer, a computer workstation, a smart phone, a tablet computer, a server, a smart camera, a smart car or other types of cellular phones, a media consumption device, a wearable device (such as a VR / AR headset, VR glasses), a set-top box, a game console, and the like.

[0017] Exemplarily, the rendering parameter can refer to all parameters input to the graphics rendering engine and required for rendering processing by the graphics rendering engine, and can include various parameters for rendering, position and color vectors of all light sources, position vectors of players or observers, sampling methods of each texture, position coordinates of each scene object, motion trajectories of moving objects, bone animation parameters, and the like, and the application does not limit this.

[0018] Exemplarily, the rendering intermediate result can refer to intermediate data generated by the graphics rendering engine for generating a to-be-displayed image (i.e., a rendered image) / video (i.e., a rendered video) in the process of generating the to-be-displayed image / video. Exemplarily, the rendering intermediate result can include but is not limited to a computer graphics motion vector (CGMV), a rendering intermediate image (the rendering intermediate image refers to an image generated before generating a final rendering image (i.e., the rendered image) described above, and the calculation complexity of the rendering intermediate image is lower than that of the rendered image; wherein the rendering intermediate image can be, for example, a rendering intermediate image without indirect lighting rendering, a rendering intermediate image without specular reflection processing, a rendering intermediate image without highlight processing, and the like), a position map, a normal map, an albedo map, a specular intensity map, a mesh ID, a material ID (wherein each material map corresponds to a Material ID), a render ID (each object (or a three-dimensional object model) corresponds to a Render ID), depth information, and the like, and the application does not limit this. Wherein the first rendering intermediate result is part of all rendering intermediate results generated in the rendering processing process. It should be noted that the type of intermediate result included in the first rendering intermediate result generated by the terminal device is the same as the type of intermediate result included in the first rendering intermediate result generated by the server, and the accuracy of the intermediate result included in the first rendering intermediate result generated by the terminal device is less than or equal to the accuracy of the intermediate result included in the first rendering intermediate result generated by the server.

[0019] It should be understood that when the server lossy encodes the residual block, the residual block parsed by the terminal device is different from the residual block encoded by the server. When the server lossless encodes the residual block, the residual block parsed by the terminal device is the same as the residual block encoded by the server.

[0020] It should be understood that the server of the present application can include more or fewer modules than those described above, and the present application does not limit this. The terminal device of the present application can include more or fewer modules than those described above, and the present application does not limit this.

[0021] It should be understood that the present application does not limit the video coding standard used by the encoder and the decoder, which can include but is not limited to: H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), H.266 / VVC (Versatile Video Coding), AV1 (AOMedia Video 1, "AOMedia" refers to a video encoding developed by the Open Media Video Alliance), and the like, as well as extension standards of these video coding standards; in addition, it can also include new video coding standards and extension standards generated with the development of video coding technology.

[0022] In a second aspect, the embodiments of the present application provide an encoding method applied to a server, the method comprising: first, rendering a three-dimensional scene based on rendering parameters to obtain a rendered image; wherein the rendering parameters include first rendering parameters obtained from a terminal device; then, selecting a first rendering intermediate result based on rendering intermediate results generated in the rendering process; thereafter, encoding the rendered image based on the first rendering intermediate result to obtain encoding data of a residual block and writing the encoding data of the residual block into a bitstream; wherein the residual block is obtained by performing residual on a to-be-encoded block in the rendered image and a corresponding prediction block, and the prediction block is obtained by performing prediction on the to-be-encoded block; the first rendering intermediate result acts on at least one processing in the encoding process: blocking, prediction or filtering; and the bitstream does not include encoding data of the first rendering intermediate result.

[0023] In this way, all rendering is performed at the terminal device, and then the server can not send the rendering intermediate result to the terminal device; therefore, the present application can reduce the interactive delay while ensuring that the code rate overhead of the data stream transmitted by the server to the terminal device is effectively reduced. In addition, since the rendering intermediate result has strong correlation with the rendered image, the present application encodes the rendered image based on the rendering intermediate result, which can ensure the image reconstruction quality.

[0024] It should be noted that the present application relates to an encoding method, which can be performed by an encoder involved in the present application. In this way, the rendering intermediate result can be fully utilized, and the encoding efficiency can be further improved.

[0025] For example, the first rendering intermediate result is part of the rendering intermediate result generated in the rendering process; for example, the first rendering intermediate result is CGMV, depth information, and Render ID, etc.

[0026] For example, in the process of encoding the rendered image, the rendered image can be first divided into blocks to obtain a plurality of to-be-encoded blocks; for a to-be-encoded block, a prediction block can be obtained by performing prediction on the to-be-encoded block based on the filtered reconstructed block; then, a residual block between the to-be-encoded block and the prediction block is determined; and then, the residual block can be encoded, and the encoding data of the residual block is written into the code stream.

[0027] For example, the residual block can be processed by transformation, quantization, and entropy encoding, etc., to obtain the encoding data of the residual block.

[0028] It should be understood that the first rendering intermediate result can also be used for other item processing such as entropy encoding in the encoding process, which is not limited in the present application.

[0029] According to the second aspect, the code stream further includes: a first indication identifier and / or a second indication identifier, wherein the first indication identifier is used to indicate whether the code stream includes the encoding data of the first rendering intermediate result; and the second indication identifier is used to indicate the type of the first rendering intermediate result. In this way, the terminal device can know whether the first rendering intermediate result is contained in the code stream, and know what type of first rendering intermediate result is generated.

[0030] For example, the first rendering intermediate result can be divided into a plurality of types, such as a motion vector type, a first image type, a second image type, etc. When the first rendering intermediate result is CGMV, the corresponding type can be the motion vector type. When the first rendering intermediate result is depth information, the corresponding type can be the first image type. When the first rendering intermediate result is Render ID, the corresponding type can be the second image type. It should be understood that the first rendering intermediate result can also include other types, which are not limited in the present application.

[0031] According to the second aspect, or any one of the implementation modes of the second aspect, the rendering parameter further includes a second rendering parameter generated by the server; and the method further includes: writing the third rendering parameter into the code stream, wherein the third rendering parameter includes all or part of the parameters in the second rendering parameter.

[0032] The rendering parameter generated by the server is more accurate than the rendering parameter generated by the terminal device. Then, the server can send part or all of the second rendering parameter to the terminal device, so that the first rendering intermediate result generated by the terminal device is more accurate, thereby improving the image quality of the image decoded based on the first rendering intermediate result.

[0033] In addition, the data amount of the second rendering parameter is small (a few / dozens of kB), which is much smaller than the rendering intermediate result. Therefore, even if the application sends the rendering parameter to the terminal device, the code rate overhead of the data stream sent by the server of the application to the terminal device is smaller than the code rate overhead of the data stream sent by the server of the prior art to the terminal device. And it can also save the computing power of the terminal device.

[0034] It should be noted that the first rendering parameter and the second rendering parameter can constitute the rendering parameter (i.e., all parameters input to the graphics rendering engine and required for rendering processing by the graphics rendering engine).

[0035] Exemplarily, the third rendering parameter can be encoded, and the encoded data of the third rendering parameter can be written into the code stream. Alternatively, the third rendering parameter can be added to the code stream without encoding, and the application does not limit this.

[0036] According to the second aspect, or any one of the implementation modes of the second aspect, the rendering parameter further includes a second rendering parameter generated by the server; the code stream further includes a third indication identifier and / or a fourth indication identifier, where: the third indication identifier is used to indicate whether the code stream includes a third rendering parameter, where the third rendering parameter includes all or part of the second rendering parameter; and the fourth indication identifier is used to indicate the type of the third rendering parameter. In this way, the terminal device can learn whether the third rendering parameter is included in the code stream, and when the third rendering parameter is part of the second rendering parameter, the terminal device can generate a fourth rendering parameter according to the type of the third rendering parameter, where the fourth rendering parameter is part of the second rendering parameter other than the third rendering parameter.

[0037] Exemplarily, the second rendering parameter can be divided into multiple types, for example, type C1 and type C2. Exemplarily, the second rendering parameter can include motion information of a rigid moving object and motion information of a non-rigid dynamic object. The type corresponding to the motion information of the rigid moving object is type C1, and the type corresponding to the motion information of the non-rigid dynamic object is type C2.

[0038] In one possible way, the third rendering parameter can include motion information of a rigid moving object and motion information of a non-rigid dynamic object.

[0039] In a possible solution, the third rendering parameter can comprise motion information of the rigid moving object. In this way, compared with the third rendering parameter comprising motion information of the rigid moving object and motion information of the non-rigid dynamic object, the code rate overhead of the data stream transmitted by the server to the terminal device can be further reduced.

[0040] In a possible solution, the third rendering parameter can comprise motion information of the non-rigid dynamic object. In this way, compared with the third rendering parameter comprising motion information of the rigid moving object and motion information of the non-rigid dynamic object, the code rate overhead of the data stream transmitted by the server to the terminal device can be further reduced.

[0041] According to the second aspect, or any possible implementation of the second aspect, the first rendering intermediate result acts on blocking in the encoding process, and the block to be encoded is a prediction unit; and the encoding of the rendered image based on the first rendering intermediate result to obtain the encoding data of the residual block comprises: blocking the rendered image based on the first rendering intermediate result to obtain a plurality of prediction units; predicting the plurality of prediction units based on the reconstructed block to obtain a plurality of prediction blocks, the plurality of prediction blocks corresponding to the plurality of prediction units in a one-to-one manner; and encoding a plurality of residual blocks between the plurality of prediction blocks and the plurality of prediction units to obtain the encoding data of the plurality of residual blocks, the plurality of residual blocks corresponding to the plurality of prediction units in a one-to-one manner.

[0042] Since the first rendering intermediate result has a strong correlation with the rendered image, the rendered image can be reasonably blocked based on the first rendering intermediate result to obtain a better prediction effect; when the prediction effect is better, the determined residual block is smaller, and the code rate can be reduced; and the reconstruction quality of the image can also be improved.

[0043] According to the second aspect, or any possible implementation of the second aspect, the first rendering intermediate result acts on prediction in the encoding process; and the encoding of the rendered image based on the first rendering intermediate result to obtain the encoding data of the residual block comprises: predicting a block to be encoded in the rendered image based on the reconstructed block and the first rendering intermediate result to obtain a prediction block corresponding to the block to be encoded; and encoding a residual block between the block to be encoded and the prediction block corresponding to the block to be encoded to obtain the encoding data of the residual block.

[0044] Since the first rendering intermediate result has a strong correlation with the rendered image, the prediction block obtained by predicting based on the first rendering intermediate result has a smaller residual block between the block to be encoded, and the code rate can be reduced.

[0045] According to a second aspect, or any possible implementation mode of the second aspect, the first rendering intermediate result acts on filtering in the encoding process; and the encoding, based on the first rendering intermediate result, of the rendered image to obtain the encoding data of the residual block comprises: predicting, based on the reconstructed block, a to-be-encoded block in the rendered image to obtain a prediction block corresponding to the to-be-encoded block, the reconstructed block being a reconstructed block filtered based on the first rendering intermediate result; and encoding a residual block between the to-be-encoded block and the prediction block corresponding to the to-be-encoded block to obtain the encoding data of the residual block.

[0046] Since the first rendering intermediate result has a strong correlation with the rendered image, the quality of the reconstructed block filtered based on the first rendering intermediate result is better; and thus, encoding, based on the filtered reconstructed block, the to-be-encoded block in the rendered image can reduce the code rate.

[0047] According to the second aspect, or any possible implementation mode of the second aspect, the first rendering intermediate result also acts on prediction in the encoding process; and the predicting, based on the reconstructed block, of the plurality of prediction units to obtain the plurality of prediction blocks comprises: predicting, based on the reconstructed block and the first rendering intermediate result, the plurality of prediction units to obtain the plurality of prediction blocks. In this way, the first rendering intermediate result acts on the blocking and the prediction in the encoding process, which can further improve the prediction effect; and thus, the code rate can be further reduced and the reconstructed quality of the image can be further improved.

[0048] According to the second aspect, or any possible implementation mode of the second aspect, the first rendering intermediate result also acts on filtering in the encoding process, and the reconstructed block is a reconstructed block filtered based on the first rendering intermediate result. In this way, the first rendering intermediate result can act on the blocking, the prediction and the filtering in the encoding process; or act on the blocking and the filtering; or act on the prediction and the filtering; and thus, the code rate can be further reduced and the reconstructed quality of the image can be further improved.

[0049] According to the second aspect, or any possible implementation mode of the second aspect, the first rendering intermediate result is depth information, and the blocking, based on the first rendering intermediate result, of the rendered image to obtain the plurality of prediction units comprises: dividing the rendered image into a plurality of coding units; generating computer graphics edge (CGE) information based on the depth information, the CGE information comprising object edge information of an object in the rendered image; and dividing the plurality of coding units based on the CGE information to obtain the plurality of prediction units.

[0050] The CGE information includes object edge information of objects in the rendered image. Based on the CGE information, the coding unit can be reasonably divided into a plurality of PUs, and then better prediction results (for example, the prediction block obtained by subsequent prediction is more accurate) can be obtained. When the prediction result is better, the determined residual block is smaller, and the code rate can be reduced. In addition, the calculation amount of dividing the PUs based on the CGE information is smaller than that of the prior art.

[0051] According to the second aspect or any one of the implementation forms of the second aspect, the first rendering intermediate result is a computer graphics motion vector (CGMV), and the CGMV is used to describe a displacement relationship between a pixel point in the rendered image and a pixel point in a reference frame of the rendered image. The method further includes: determining, from the reference frame of the rendered image, a reconstructed block matched with the to-be-encoded block; and generating the prediction block based on the CGMV and the reconstructed block matched with the to-be-encoded block.

[0052] The CGMV is generated by using a graphics method, and thus the problem of inaccurate motion estimation of an existing encoder is avoided. The CGMV is a pixel-level MV, while the MV generated by the existing encoder is an image block-level MV. The pixel-level MV can more accurately depict the edge of an object and reduce prediction error. Therefore, the prediction block is determined based on the CGMV, which can reduce the error of the prediction block to a certain extent, improve the accuracy of the prediction block, and then improve the inter-frame coding compression efficiency.

[0053] According to the second aspect or any one of the implementation forms of the second aspect, the first rendering intermediate result is a rendering identifier (Render ID), and the Render ID is used to describe an object to which a pixel point in the reconstructed block belongs. The method further includes: determining a filtering parameter based on the Render ID; and filtering the reconstructed block based on the filtering parameter. Compared with the prior art, the filtering parameter is determined by determining whether the pixels on both sides of the boundary line of the reconstructed block belong to the same object based on the Render ID, and the reconstructed block is filtered based on the filtering parameter, which can better reduce the blocking effect between the reconstructed blocks, and thus the quality of the reconstructed block can be improved. In this way, the to-be-encoded block in the rendered image is encoded with reference to the filtered reconstructed block, and the code rate can be reduced.

[0054] According to the second aspect or any one of the implementation forms of the second aspect, the CGE information is encoded, and the encoded data of the CGE information is written into a bitstream. In this way, the terminal device can determine the block information based on the CGE information without generating the CGE information, and the computing power of the terminal device can be reduced.

[0055] In a third aspect, an embodiment of the present application provides a decoding method, which comprises the following steps: first, receiving a code stream; then, parsing the code stream to obtain a parsing result, the parsing result comprising a residual block corresponding to a current frame; thereafter, performing rendering processing on a three-dimensional scene based on a rendering parameter corresponding to the current frame, and generating a first rendering intermediate result in the rendering processing; wherein the rendering parameter comprises a first rendering parameter generated by a terminal device; subsequently, reconstructing based on the first rendering intermediate result and the residual block to obtain a reconstructed image of the current frame; wherein the first rendering intermediate result is used for at least one processing in the reconstruction process: prediction or filtering.

[0056] It should be noted that the present application relates to a decoding method, which can be performed by an encoder related to the present application.

[0057] According to the third aspect, the parsing result further comprises a first indication identifier and a second indication identifier, the first indication identifier being used for indicating whether the code stream comprises encoded data of the first rendering intermediate result generated by the server, and the second indication identifier being used for indicating a type of the first rendering intermediate result generated by the server; and the first rendering intermediate result is generated in the rendering processing comprises: when it is determined based on the first indication identifier that the code stream does not comprise the encoded data of the first rendering intermediate result generated by the server, the first rendering intermediate result is generated in the rendering processing based on the second indication identifier.

[0058] According to the third aspect, or any one of the third aspect, the third aspect has the following implementation manners, the parsing result further comprises a third indication identifier and a fourth indication identifier; the third indication identifier is used for indicating whether the code stream comprises a third rendering parameter, and the fourth indication identifier is used for indicating a type of the third rendering parameter; the method further comprises: when it is determined based on the third indication identifier that the code stream comprises the third rendering parameter, the type of the third rendering parameter is determined based on the fourth indication identifier; a fourth rendering parameter of another type is generated based on the type of the third rendering parameter; the other type is a type other than the type of the third rendering parameter in a plurality of types corresponding to a second rendering parameter generated by the server; wherein the rendering parameter further comprises the third rendering parameter and the fourth rendering parameter.

[0059] In this way, when the third rendering parameter of the server to the terminal device is part of the second rendering parameter, the terminal device can generate other rendering parameters of the second rendering parameter other than the third rendering parameter according to the fourth indication identifier in the code stream, so as to obtain all rendering parameters required by the graphic rendering engine for rendering processing.

[0060] For example, when it is determined based on the fourth indication identifier that the third rendering parameter is the entire second rendering parameter, the rendering parameter further comprises the third rendering parameter.

[0061] It should be understood that when the server uses lossy encoding for the third rendering parameter, the third rendering parameter parsed by the terminal device will differ from the server's encoded third rendering parameter. When the server uses lossless encoding for the third rendering parameter, the third rendering parameter parsed by the terminal device will be the same as the server's encoded third rendering parameter.

[0062] According to the third aspect, or any implementation of the third aspect above, the method further includes: when it is determined based on the third indicator that the bitstream does not include the third rendering parameter, the terminal device generates the second rendering parameter corresponding to the current frame; wherein the rendering parameter also includes the second rendering parameter generated by the terminal device.

[0063] In this way, even if the server does not send the second rendering parameters to the terminal device, the terminal device can still generate the second rendering parameters to obtain all the rendering parameters required by the graphics rendering engine for rendering processing.

[0064] It should be understood that the parameter types contained in the second rendering parameters generated by the terminal device are the same as those contained in the second rendering parameters generated by the server, and the precision of the parameters contained in the second rendering parameters generated by the terminal device is less than or equal to the precision of the parameters contained in the second rendering parameters generated by the server.

[0065] According to the third aspect, or any implementation of the third aspect above, there are multiple residual blocks, and the first intermediate rendering result is used for prediction in the reconstruction process; reconstruction is performed based on the first intermediate rendering result and the residual blocks to obtain the reconstructed image of the current frame, including: determining the block information of the current frame based on the first intermediate rendering result; predicting multiple prediction units included in the current frame based on the block information and the reconstructed blocks to obtain multiple prediction blocks, with multiple prediction blocks corresponding one-to-one with multiple prediction units, and multiple residual blocks corresponding one-to-one with multiple prediction units; and determining the reconstructed image of the current frame based on the multiple residual blocks and the multiple prediction blocks.

[0066] According to the third aspect, or any implementation of the third aspect above, the first rendering intermediate result is applied to the prediction in the reconstruction process; reconstruction is performed based on the first rendering intermediate result and the residual block to obtain the reconstructed image of the current frame, including: predicting the block to be decoded in the current frame based on the first rendering intermediate result and the reconstructed block to obtain the prediction block corresponding to the block to be decoded; and determining the reconstructed image of the current frame based on the residual block corresponding to the block to be decoded in the residual block corresponding to the current frame and the prediction block corresponding to the block to be decoded.

[0067] According to a third aspect, or any possible implementation of the third aspect, the filtering in the reconstruction process is based on the first rendering intermediate result; and the reconstructing based on the first rendering intermediate result and the residual block to obtain the reconstructed image of the current frame comprises: predicting a to-be-decoded block in the current frame based on a reconstructed block to obtain a prediction block corresponding to the to-be-decoded block, the reconstructed block being a reconstructed block filtered based on the first rendering intermediate result; and determining the reconstructed image of the current frame based on a residual block corresponding to the to-be-decoded block in the residual block corresponding to the current frame and the prediction block corresponding to the to-be-decoded block.

[0068] According to a third aspect, or any possible implementation of the third aspect, the predicting a plurality of prediction units included in the current frame based on the partition information and the reconstructed block to obtain a plurality of prediction blocks comprises: predicting the plurality of prediction units included in the current frame based on the partition information, the first rendering intermediate result and the reconstructed block to obtain the plurality of prediction blocks.

[0069] According to a third aspect, or any possible implementation of the third aspect, the first rendering intermediate result also acts on filtering in the reconstruction process, and the reconstructed block is a reconstructed block filtered based on the first rendering intermediate result.

[0070] According to a third aspect, or any possible implementation of the third aspect, the first rendering intermediate result is depth information, and the determining the partition information of the current frame based on the first rendering intermediate result comprises: generating computer graphics edge (CGE) information based on the depth information, the CGE information comprising object edge information of an object in the current frame; and determining the partition information of the current frame based on the CGE information.

[0071] According to a third aspect, or any possible implementation of the third aspect, the first rendering intermediate result is a computer graphics motion vector (CGMV), the CGMV being used to describe a displacement relationship between a pixel point of the current frame and a pixel point of a reference frame of the current frame, and the predicting a to-be-decoded block in the current frame based on the first rendering intermediate result and the reconstructed block to obtain a prediction block corresponding to the to-be-decoded block comprises: determining a reconstructed block matching the to-be-decoded block from the reference frame of the current frame; and generating the prediction block corresponding to the to-be-decoded block based on the CGMV and the reconstructed block matching the to-be-decoded block.

[0072] According to a third aspect, or any possible implementation of the third aspect, the first rendering intermediate result is a rendering identifier (Render ID), the Render ID being used to describe an object to which a pixel point in the reconstructed block belongs; and the method further comprises: determining a filtering parameter based on the Render ID; and filtering the reconstructed block based on the filtering parameter.

[0073] According to a third aspect, or any possible implementation of the third aspect, the residual blocks are multiple, and the first rendering intermediate result acts on prediction in the reconstruction process; the decoding result further comprises CGE information, and the CGE information comprises object edge information of an object in the current frame; the reconstruction of the current frame based on the first rendering intermediate result and the residual blocks comprises: determining block information of the current frame based on the CGE information; predicting multiple prediction units included in the current frame based on the block information and the reconstruction blocks to obtain multiple prediction blocks, the multiple prediction blocks correspond to the multiple prediction units one by one, and the multiple residual blocks correspond to the multiple prediction units one by one; and determining the reconstruction image of the current frame based on the multiple residual blocks and the multiple prediction blocks.

[0074] The third aspect and any possible implementation of the third aspect correspond to the second aspect and any possible implementation of the second aspect respectively. The technical effects of the third aspect and any possible implementation of the third aspect correspond to the technical effects of the second aspect and any possible implementation of the second aspect, which are not described herein again.

[0075] According to a fourth aspect, the present application further provides an end-cloud cooperative system, comprising: a server and a terminal device, the server comprising a first rendering module, an encoder and a first communication module, and the terminal device comprising a second communication module, a second rendering module and a decoder; wherein:

[0076] The first rendering module is configured to perform rendering processing on a three-dimensional scene based on rendering parameters to obtain a rendered image; wherein the rendering parameters comprise first rendering parameters obtained from the terminal device; and the first rendering intermediate result is selected based on rendering intermediate results generated in the rendering processing.

[0077] The encoder is configured to encode the rendered image based on the first rendering intermediate result to obtain encoding data of the residual blocks and write the encoding data of the residual blocks into a bitstream; and encode the second rendering intermediate result and write encoding data of the second rendering intermediate result into the bitstream; wherein the residual blocks are obtained by performing residual calculation on a to-be-encoded block and a corresponding prediction block in the rendered image, and the prediction block is obtained by performing prediction on the to-be-encoded block; the first rendering intermediate result acts on at least one processing in the encoding process, including blocking, prediction or filtering; and the second rendering intermediate result is part of the first rendering intermediate result.

[0078] The first communication module is configured to send the bitstream.

[0079] The second communication module is configured to receive the bitstream.

[0080] The decoder is configured to parse the bitstream to obtain a parsing result, the parsing result comprising residual blocks corresponding to the current frame and a second rendering intermediate result corresponding to the current frame.

[0081] a second rendering module, configured to perform rendering processing on the three-dimensional scene based on a rendering parameter corresponding to the current frame, and generate a third rendering intermediate result based on a second rendering intermediate result corresponding to the current frame during the rendering processing, wherein the rendering parameter corresponding to the current frame comprises the first rendering parameter generated by the terminal device;

[0082] the decoder is further configured to perform reconstruction based on the first rendering intermediate result and a residual block corresponding to the current frame to obtain a reconstructed image of the current frame, wherein the first rendering intermediate result used to determine the reconstructed image of the current frame comprises the second rendering intermediate result and the third rendering intermediate result, and the first rendering intermediate result used to determine the reconstructed image of the current frame is used for at least one processing in a reconstruction process: prediction or filtering.

[0083] In this way, part of the rendering is performed by the terminal device, and then the server can only send part of the rendering intermediate result to the terminal device, so that the application can reduce the interactive delay while effectively reducing the code rate overhead of the data stream transmitted by the server to the terminal device. In addition, since the rendering intermediate result has strong correlation with the rendered image, the application encodes the rendered image based on the rendering intermediate result, so that the image reconstruction quality can be ensured.

[0084] In addition, the encoder involved in the application is a modified (or optimized) encoder of an existing encoder, which can encode the rendered image based on the first rendering intermediate result. In other words, the encoder involved in the application can encode the rendered image based on the first rendering intermediate result, and includes all or part of the functions of the existing encoder. The decoder involved in the application is a modified (or optimized) decoder of an existing decoder, which can decode based on the first rendering intermediate result. In other words, the decoder involved in the application can decode based on the first rendering intermediate result, and includes all or part of the functions of the existing decoder. In this way, the rendering intermediate result can be fully utilized to further improve the coding efficiency.

[0085] For example, assuming that the first rendering intermediate result is CGMV, the CGMV can include: CGMV of a static object, CGMV of a rigid dynamic object and CGMV of a non-rigid dynamic object; and the second rendering intermediate result can include: any one or two of the CGMV of the static object, the CGMV of the rigid dynamic object and the CGMV of the non-rigid dynamic object.

[0086] In one possible way, the server can encode the CGMV of the static object. In this way, the terminal device does not need to calculate the CGMV of the static object, thereby saving the computing power of the terminal device.

[0087] In a possible implementation, the server can encode the CGMV of the dynamic object. In this way, the terminal device does not need to calculate the CGMV of the dynamic object, and the computing power of the terminal device is saved. In addition, the calculation amount of the CGMV of the dynamic object is larger than that of the CGMV of the static object, and therefore, the CGMV of the dynamic object is encoded into the second code stream and sent to the terminal device, so that more computing power of the terminal device is saved.

[0088] For example, the CGMV of the dynamic object can include the CGMV of a rigid dynamic object and / or the CGMV of a non-rigid dynamic object.

[0089] In a possible implementation, the CGMV of the rigid dynamic object and the CGMV of the non-rigid dynamic object can be encoded.

[0090] In a possible implementation, the CGMV of the rigid dynamic object can be encoded. In this way, compared with encoding the CGMV of the rigid dynamic object and the CGMV of the non-rigid dynamic object, the code rate can be reduced.

[0091] In a possible implementation, the CGMV of the non-rigid dynamic object can be encoded. In this way, compared with encoding the CGMV of the rigid dynamic object and the CGMV of the non-rigid dynamic object, the code rate can be reduced. In addition, because the number of triangular facets of the non-rigid dynamic object is large, the calculation amount of the CGMV of the non-rigid dynamic object is large, and therefore, compared with encoding the CGMV of the rigid dynamic object, encoding the CGMV of the non-rigid dynamic object can save more computing power of the terminal device.

[0092] For example, assuming that the first rendering intermediate result is depth information (that is, a depth map), the server can take part of the image in the depth map as the second rendering intermediate result.

[0093] For example, assuming that the first rendering intermediate result is a Render ID (that is, a Render ID map), the server can take part of the image in the Render ID map as the second rendering intermediate result.

[0094] In a fifth aspect, the embodiments of the present application further provide an encoding method, which comprises: first, performing rendering processing on a three-dimensional scene based on rendering parameters to obtain a rendered image; wherein the rendering parameters comprise first rendering parameters obtained from a terminal device; then, selecting a first rendering intermediate result based on rendering intermediate results generated in the rendering processing; subsequently, encoding the rendered image based on the first rendering intermediate result to obtain encoding data of a residual block and write the encoding data of the residual block into a bitstream; and encoding a second rendering intermediate result and write encoding data of the second rendering intermediate result into the bitstream. The residual block is obtained by performing residual calculation on a to-be-encoded block and a corresponding prediction block in the rendered image, and the prediction block is obtained by performing prediction on the to-be-encoded block; the first rendering intermediate result is used for at least one processing in the encoding process: blocking, prediction or filtering; and the second rendering intermediate result is part of the first rendering intermediate result.

[0095] In this way, part of the rendering is performed at the terminal device, and then the server can only send part of the rendering intermediate result to the terminal device; therefore, the present application can reduce the interactive delay while ensuring that the code rate overhead of the data stream transmitted by the server to the terminal device is effectively reduced. In addition, since the rendering intermediate result has strong correlation with the rendered image, the present application encodes the rendered image based on the rendering intermediate result, which can ensure the image reconstruction quality.

[0096] It should be noted that the present application relates to an encoding method, which can be performed by an encoder related to the present application. In this way, the rendering intermediate result can be fully utilized to further improve the encoding efficiency.

[0097] According to the fifth aspect, the bitstream further comprises: a first indication identifier and / or a second indication identifier, wherein the first indication identifier is used to indicate whether the bitstream comprises the encoding data of the second rendering intermediate result; and the second indication identifier is used to indicate the type and the sub-type of the second rendering intermediate result. In this way, the terminal device can know what type and sub-type of third rendering intermediate result is generated, wherein the third rendering intermediate result is part of the first rendering intermediate result other than the second rendering intermediate result.

[0098] For example, the second rendering intermediate result can also comprise multiple types, such as a motion vector type, a first image type, a second image type, and the like. When the second rendering intermediate result is CGMV, the corresponding type can be the motion vector type. When the second rendering intermediate result is depth information, the corresponding type can be the first image type. When the second rendering intermediate result is Render ID, the corresponding type can be the second image type.

[0099] For example, each type of the second rendering intermediate result can comprise multiple sub-types.

[0100] For example, the motion vector class can include: a sub-type A1, a sub-type A2 and a sub-type A3; wherein the CGMV of the static object corresponds to the sub-type A1, the CGMV of the rigid dynamic object corresponds to the sub-type A2, and the CGMV of the non-rigid dynamic object corresponds to the sub-type A3.

[0101] For example, the first image class can include: a sub-type B1 and a sub-type B2. The sub-type corresponding to the region corresponding to the four corners in the depth map is the sub-type B1, and the sub-type corresponding to the region other than the region corresponding to the four corners in the depth map is the sub-type B2.

[0102] For example, the second image class can include: a sub-type B3 and a sub-type B4. The sub-type corresponding to the region corresponding to the four corners in the Render ID map is the sub-type B3, and the sub-type corresponding to the region other than the region corresponding to the four corners in the Render ID map is the sub-type B4.

[0103] According to a fifth aspect, or any possible implementation mode of the fifth aspect, the rendering parameter further includes a second rendering parameter generated by the server; and the method further includes: writing the third rendering parameter into the code stream, wherein the third rendering parameter includes all or part of the parameters in the second rendering parameter.

[0104] According to a fifth aspect, or any possible implementation mode of the fifth aspect, the rendering parameter further includes a second rendering parameter generated by the server; and the code stream further includes: a third indication identifier and / or a fourth indication identifier, wherein: the third indication identifier is used to indicate whether the code stream includes a third rendering parameter, wherein the third rendering parameter includes all or part of the parameters in the second rendering parameter; and the fourth indication identifier is used to indicate the type of the third rendering parameter. In this way, the terminal device can learn whether the third rendering parameter is included in the code stream; and when the third rendering parameter is part of the second rendering parameter, the terminal device can generate a fourth rendering parameter according to the type of the third rendering parameter, wherein the fourth rendering parameter is part of the second rendering parameter other than the third rendering parameter.

[0105] According to a fifth aspect, or any possible implementation mode of the fifth aspect, the first rendering intermediate result acts on a block in the encoding process, the block to be encoded is a prediction unit, and the encoding of the rendered image based on the first rendering intermediate result to obtain the encoding data of the residual block includes: blocking the rendered image based on the first rendering intermediate result to obtain a plurality of prediction units; predicting the plurality of prediction units based on the reconstructed block to obtain a plurality of prediction blocks, the plurality of prediction blocks and the plurality of prediction units corresponding one by one; and encoding a plurality of residual blocks between the plurality of prediction blocks and the plurality of prediction units to obtain encoding data of the plurality of residual blocks, the plurality of residual blocks and the plurality of prediction units corresponding one by one.

[0106] According to a fifth aspect, or any implementation mode of the fifth aspect, the first rendering intermediate result acts on prediction in the encoding process, and the rendered image is encoded based on the first rendering intermediate result to obtain the encoding data of the residual block, including: predicting the to-be-encoded block in the rendered image based on the reconstructed block and the first rendering intermediate result to obtain a prediction block corresponding to the to-be-encoded block; and encoding a residual block between the to-be-encoded block and the prediction block corresponding to the to-be-encoded block to obtain the encoding data of the residual block.

[0107] According to a fifth aspect, or any implementation mode of the fifth aspect, the first rendering intermediate result acts on filtering in the encoding process, and the rendered image is encoded based on the first rendering intermediate result to obtain the encoding data of the residual block, including: predicting the to-be-encoded block in the rendered image based on the reconstructed block to obtain a prediction block corresponding to the to-be-encoded block, the reconstructed block being a reconstructed block filtered based on the first rendering intermediate result; and encoding a residual block between the to-be-encoded block and the prediction block corresponding to the to-be-encoded block to obtain the encoding data of the residual block.

[0108] According to a fifth aspect, or any implementation mode of the fifth aspect, the first rendering intermediate result further acts on prediction in the encoding process, and a plurality of prediction blocks are predicted based on the reconstructed block to obtain the plurality of prediction blocks, including: predicting the plurality of prediction units based on the reconstructed block and the first rendering intermediate result to obtain the plurality of prediction blocks.

[0109] According to a fifth aspect, or any implementation mode of the fifth aspect, the first rendering intermediate result further acts on filtering in the encoding process, and the reconstructed block is a reconstructed block filtered based on the first rendering intermediate result.

[0110] According to a fifth aspect, or any implementation mode of the fifth aspect, the first rendering intermediate result is depth information, and the rendered image is blocked based on the first rendering intermediate result to obtain a plurality of prediction units, including: dividing the rendered image into a plurality of coding units; generating computer graphics edge (CGE) information based on the depth information, the CGE information including object edge information of an object in the rendered image; and dividing the plurality of coding units based on the CGE information to obtain the plurality of prediction units.

[0111] According to a fifth aspect, or any possible implementation mode of the fifth aspect, the first rendering intermediate result is a computer graphics motion vector (CGMV), the CGMV being used to describe a displacement relationship between a pixel point in the rendered image and a pixel point in a reference frame of the rendered image; the predicting the to-be-encoded block in the rendered image based on the reconstructed block and the first rendering intermediate result to obtain a prediction block corresponding to the to-be-encoded block comprises: determining the reconstructed block matched with the to-be-encoded block from the reference frame of the rendered image; and generating the prediction block based on the CGMV and the reconstructed block matched with the to-be-encoded block.

[0112] According to a fifth aspect, or any possible implementation mode of the fifth aspect, the first rendering intermediate result is a rendering identifier (Render ID), the Render ID being used to describe an object to which a pixel point in the reconstructed block belongs; the method further comprises: determining a filtering parameter based on the Render ID; and filtering the reconstructed block based on the filtering parameter.

[0113] According to a fifth aspect, or any possible implementation mode of the fifth aspect, the CGE information is encoded, and encoded data of the CGE information is written into a bitstream.

[0114] The fifth aspect and any possible implementation mode of the fifth aspect correspond to the second aspect and any possible implementation mode of the second aspect respectively. For details of the technical effects of the fifth aspect and any possible implementation mode of the fifth aspect, refer to the technical effects of the second aspect and any possible implementation mode of the second aspect, which will not be described here.

[0115] According to a sixth aspect, the present application further provides a decoding method applied to a terminal device, the decoding method comprising the following steps: first, receiving a bitstream; then, parsing the bitstream to obtain a parsing result, the parsing result comprising a residual block corresponding to a current frame and a second rendering intermediate result corresponding to the current frame; next, performing rendering processing on a three-dimensional scene based on a rendering parameter corresponding to the current frame, and generating a third rendering intermediate result based on the second rendering intermediate result during the rendering processing, the rendering parameter comprising a first rendering parameter generated by the terminal device; subsequently, reconstructing based on a first rendering intermediate result and the residual block to obtain a reconstructed image of the current frame; wherein the first rendering intermediate result comprises the second rendering intermediate result and the third rendering intermediate result, and the first rendering intermediate result is used for at least one processing in a reconstruction process: prediction or filtering.

[0116] In this way, the terminal device obtains the first rendering intermediate result, which includes the third rendering intermediate result generated by the terminal device and the second rendering intermediate result generated by the server. Since the second rendering intermediate result generated by the server is more accurate than the second rendering intermediate result generated by the terminal device, the first rendering intermediate result obtained is more accurate, thereby improving the image quality of the reconstructed image. In addition, the terminal device only needs to generate part of the first rendering intermediate result, which can save the computing power of the terminal device and improve the decoding efficiency.

[0117] It should be understood that when the server lossy encodes the second intermediate result, the second intermediate result obtained by the terminal device is different from the second intermediate result encoded by the server. When the server lossless encodes the second intermediate result, the second intermediate result obtained by the terminal device is the same as the second intermediate result encoded by the server.

[0118] It should be understood that since the third rendering intermediate result is generated by the terminal device, the type of the intermediate result included in the first rendering intermediate result obtained by the terminal device is the same as the type of the intermediate result included in the first rendering intermediate result generated by the server. The accuracy of the intermediate result included in the first rendering intermediate result obtained by the terminal device is less than or equal to the accuracy of the intermediate result included in the first rendering intermediate result generated by the server.

[0119] It should be noted that the present application relates to a decoding method, which can be performed by an encoder related to the present application.

[0120] According to a sixth aspect, the decoding result further includes a second indication identifier, the second indication identifier being used to indicate the type and the sub-type of the second rendering intermediate result. In the rendering processing process, the third rendering intermediate result is generated based on the second rendering intermediate result, including: based on the second indication identifier, determining the type and the sub-type corresponding to the second rendering intermediate result; in the rendering processing process, the third rendering intermediate result of other sub-types is generated based on the type and the sub-type corresponding to the second rendering intermediate result; wherein the other sub-types are sub-types in the sub-types included in the type corresponding to the second rendering intermediate result, except for the sub-type corresponding to the second rendering intermediate result.

[0121] According to a sixth aspect, or any possible implementation mode of the sixth aspect, the parsing result further comprises a third indication identifier and a fourth indication identifier; the third indication identifier is used to indicate whether the code stream comprises a third rendering parameter, and the fourth indication identifier is used to indicate a type of the third rendering parameter; the method further comprises: when it is determined, based on the third indication identifier, that the code stream comprises the third rendering parameter, determining, based on the fourth indication identifier, the type of the third rendering parameter; generating a fourth rendering parameter of another type based on the type of the third rendering parameter; the other type is a type other than the type of the third rendering parameter in a plurality of types corresponding to the second rendering parameter generated by the server; and the rendering parameters further comprise the third rendering parameter and the fourth rendering parameter.

[0122] According to the sixth aspect, or any possible implementation mode of the sixth aspect, the method further comprises: when it is determined, based on the third indication identifier, that the code stream does not comprise the third rendering parameter, generating, by the terminal device, the second rendering parameter corresponding to the current frame; and the rendering parameters further comprise the second rendering parameter generated by the terminal device.

[0123] According to the sixth aspect, or any possible implementation mode of the sixth aspect, the residual blocks are multiple, and the first rendering intermediate result is used for prediction in a reconstruction process; the reconstruction based on the first rendering intermediate result and the residual blocks to obtain the reconstructed image of the current frame comprises: determining, based on the first rendering intermediate result, block information of the current frame; predicting, based on the block information and the reconstruction blocks, a plurality of prediction units included in the current frame to obtain a plurality of prediction blocks, the plurality of prediction blocks and the plurality of prediction units correspond one by one, and the plurality of residual blocks and the plurality of prediction units correspond one by one; and determining the reconstructed image of the current frame based on the plurality of residual blocks and the plurality of prediction blocks.

[0124] According to the sixth aspect, or any possible implementation mode of the sixth aspect, the first rendering intermediate result is used for prediction in a reconstruction process; the reconstruction based on the first rendering intermediate result and the residual blocks to obtain the reconstructed image of the current frame comprises: predicting, based on the first rendering intermediate result and the reconstruction blocks, a to-be-decoded block in the current frame to obtain a prediction block corresponding to the to-be-decoded block; and determining the reconstructed image of the current frame based on a residual block corresponding to the to-be-decoded block in the residual blocks corresponding to the current frame and the prediction block corresponding to the to-be-decoded block.

[0125] According to the sixth aspect, or any possible implementation mode of the sixth aspect, the first rendering intermediate result is used for filtering in a reconstruction process; the reconstruction based on the first rendering intermediate result and the residual blocks to obtain the reconstructed image of the current frame comprises: predicting, based on the reconstruction blocks, a to-be-decoded block in the current frame to obtain a prediction block corresponding to the to-be-decoded block, the reconstruction blocks being the reconstruction blocks filtered based on the first rendering intermediate result; and determining the reconstructed image of the current frame based on a residual block corresponding to the to-be-decoded block in the residual blocks corresponding to the current frame and the prediction block corresponding to the to-be-decoded block.

[0126] According to a sixth aspect, or any possible implementation mode of the sixth aspect, the predicting the plurality of prediction units included in the current frame based on the blocking information and the reconstructed blocks to obtain a plurality of prediction blocks comprises: predicting the plurality of prediction units included in the current frame based on the blocking information, the first rendering intermediate result and the reconstructed blocks to obtain the plurality of prediction blocks.

[0127] According to a sixth aspect, or any possible implementation mode of the sixth aspect, the first rendering intermediate result further acts on filtering in the reconstruction process, and the reconstructed blocks are reconstructed blocks filtered based on the first rendering intermediate result.

[0128] According to a sixth aspect, or any possible implementation mode of the sixth aspect, the first rendering intermediate result is depth information, and the blocking information of the current frame is determined based on the first rendering intermediate result, comprising: generating computer graphics edge (CGE) information based on the depth information, the CGE information comprising object edge information of objects in the current frame; and determining the blocking information of the current frame based on the CGE information.

[0129] According to a sixth aspect, or any possible implementation mode of the sixth aspect, the first rendering intermediate result is a computer graphics motion vector (CGMV), the CGMV being used to describe a displacement relationship between a pixel point of the current frame and a pixel point of a reference frame of the current frame, and the to-be-decoded block in the current frame is predicted based on the first rendering intermediate result and the reconstructed blocks to obtain a prediction block corresponding to the to-be-decoded block, comprising: determining a reconstructed block matched with the to-be-decoded block from the reference frame of the current frame; and generating the prediction block corresponding to the to-be-decoded block based on the CGMV and the reconstructed block matched with the to-be-decoded block.

[0130] According to a sixth aspect, or any possible implementation mode of the sixth aspect, the first rendering intermediate result is a rendering identifier (Render ID), the Render ID being used to describe an object to which a pixel point in the reconstructed block belongs; and the method further comprises: determining a filtering parameter based on the Render ID; and filtering the reconstructed block based on the filtering parameter.

[0131] According to a sixth aspect, or any possible implementation mode of the sixth aspect, the residual blocks are a plurality of residual blocks, the first rendering intermediate result acts on prediction in the reconstruction process; the decoding result further comprises CGE information, the CGE information comprising object edge information of objects in the current frame; and the current frame is reconstructed based on the first rendering intermediate result and the residual blocks to obtain a reconstructed image of the current frame, comprising: determining blocking information of the current frame based on the CGE information; predicting a plurality of prediction units included in the current frame based on the blocking information and the reconstructed blocks to obtain a plurality of prediction blocks, the plurality of prediction blocks corresponding to the plurality of prediction units one by one, and the plurality of residual blocks corresponding to the plurality of prediction units one by one; and determining the reconstructed image of the current frame based on the plurality of residual blocks and the plurality of prediction blocks.

[0132] The sixth aspect and any possible implementation manner of the sixth aspect correspond to the third aspect and any possible implementation manner of the third aspect respectively. The technical effects of the sixth aspect and any possible implementation manner of the sixth aspect correspond to the technical effects of the third aspect and any possible implementation manner of the third aspect, which will not be described here.

[0133] In a seventh aspect, the present application provides a code stream generation method, which generates a code stream based on the first rendering parameter of the third aspect and any possible implementation manner of the third aspect.

[0134] In an eighth aspect, the present application provides a code stream generation method, which generates a code stream based on the first rendering parameter of the sixth aspect and any possible implementation manner of the sixth aspect.

[0135] In a ninth aspect, the present application provides a server, which comprises a memory and a processor, wherein the memory is coupled to the processor; and the memory stores program instructions, which, when executed by the processor, cause the server to perform the encoding method in the second aspect or any possible implementation manner of the second aspect; or cause the server to perform the encoding method in the fifth aspect or any possible implementation manner of the fifth aspect.

[0136] The ninth aspect and any possible implementation manner of the ninth aspect correspond to the second aspect and any possible implementation manner of the second aspect respectively; or correspond to the fifth aspect and any possible implementation manner of the fifth aspect respectively. The technical effects of the ninth aspect and any possible implementation manner of the ninth aspect correspond to the technical effects of the second aspect and any possible implementation manner of the second aspect, or correspond to the technical effects of the fifth aspect and any possible implementation manner of the fifth aspect, which will not be described here.

[0137] In a tenth aspect, the present application provides a terminal device, which comprises a memory and a processor, wherein the memory is coupled to the processor; and the memory stores program instructions, which, when executed by the processor, cause the terminal device to perform the decoding method in the third aspect or any possible implementation manner of the third aspect; or cause the terminal device to perform the decoding method in the sixth aspect or any possible implementation manner of the sixth aspect.

[0138] The tenth aspect and any possible implementation manner of the tenth aspect correspond to the third aspect and any possible implementation manner of the third aspect respectively, or correspond to the sixth aspect and any possible implementation manner of the sixth aspect respectively. For details, refer to the technical effects of the second aspect and any possible implementation manner of the second aspect, or refer to the technical effects of the sixth aspect and any possible implementation manner of the sixth aspect, which will not be described herein again.

[0139] In a eleventh aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of a server and send a signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the server executes the encoding method in the second aspect or any possible implementation manner of the second aspect; or the server executes the encoding method in the fifth aspect or any possible implementation manner of the fifth aspect.

[0140] The eleventh aspect and any possible implementation manner of the eleventh aspect correspond to the second aspect and any possible implementation manner of the second aspect respectively, or correspond to the fifth aspect and any possible implementation manner of the fifth aspect respectively. For details, refer to the technical effects of the second aspect and any possible implementation manner of the second aspect, or refer to the technical effects of the fifth aspect and any possible implementation manner of the fifth aspect, which will not be described herein again.

[0141] In a twelfth aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of a terminal device and send a signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the terminal device executes the decoding method in the third aspect or any possible implementation manner of the third aspect, or the terminal device executes the decoding method in the sixth aspect or any possible implementation manner of the sixth aspect.

[0142] The twelfth aspect and any possible implementation manner of the twelfth aspect correspond to the third aspect and any possible implementation manner of the third aspect respectively, or correspond to the sixth aspect and any possible implementation manner of the sixth aspect respectively. For details, refer to the technical effects of the third aspect and any possible implementation manner of the third aspect, or refer to the technical effects of the sixth aspect and any possible implementation manner of the sixth aspect, which will not be described herein again.

[0143] In a thirteenth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to execute the encoding method in the second aspect or any possible implementation thereof; or causes the computer or processor to execute the encoding method in the fifth aspect or any possible implementation thereof.

[0144] The thirteenth aspect and any implementation thereof correspond to the second aspect and any implementation thereof, respectively; or, respectively, to the fifth aspect and any implementation thereof. The technical effects corresponding to the thirteenth aspect and any implementation thereof can be found in the technical effects corresponding to the second aspect and any implementation thereof, or in the technical effects corresponding to the fifth aspect and any implementation thereof, and will not be repeated here.

[0145] In a fourteenth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to perform a decoding method in the third aspect or any possible implementation thereof, or causes the computer or processor to perform a decoding method in the sixth aspect or any possible implementation thereof.

[0146] The fourteenth aspect and any implementation thereof correspond to the third aspect and any implementation thereof, or to the sixth aspect and any implementation thereof. The technical effects corresponding to the fourteenth aspect and any implementation thereof can be found in the technical effects corresponding to the third aspect and any implementation thereof, or in the technical effects corresponding to the sixth aspect and any implementation thereof, and will not be repeated here.

[0147] In a fifteenth aspect, embodiments of this application provide a computer program product including computing instructions that, when executed by a computer or processor, cause the computer or processor to perform the encoding method in the second aspect or any possible implementation thereof, or cause the computer or processor to perform the encoding method in the fifth aspect or any possible implementation thereof.

[0148] The fifteenth aspect and any possible implementation manner of the fifteenth aspect correspond to the second aspect and any possible implementation manner of the second aspect respectively, or correspond to the fifth aspect and any possible implementation manner of the fifth aspect respectively. For details, refer to the technical effects of the second aspect and any possible implementation manner of the second aspect, or refer to the technical effects of the fifth aspect and any possible implementation manner of the fifth aspect, which will not be described herein again.

[0149] In a sixteenth aspect, an embodiment of the present application provides a computer program product, which includes computing instructions. When the computing instructions are executed by a computer or a processor, the computer or the processor executes the decoding method in the third aspect or any possible implementation manner of the third aspect, or executes the decoding method in the sixth aspect or any possible implementation manner of the sixth aspect.

[0150] The sixteenth aspect and any possible implementation manner of the sixteenth aspect correspond to the third aspect and any possible implementation manner of the third aspect respectively, or correspond to the sixth aspect and any possible implementation manner of the sixth aspect respectively. For details, refer to the technical effects of the third aspect and any possible implementation manner of the third aspect, or refer to the technical effects of the sixth aspect and any possible implementation manner of the sixth aspect, which will not be described herein again.

[0151] In a seventeenth aspect, an embodiment of the present application provides a device for storing a code stream, which includes a receiver and at least one storage medium. The receiver is configured to receive the code stream. The at least one storage medium is configured to store the code stream. The code stream is generated according to the second aspect and any possible implementation manner of the second aspect, or generated according to the fifth aspect and any possible implementation manner of the fifth aspect, or generated according to the seventh aspect, or generated according to the eighth aspect.

[0152] The seventeenth aspect and any possible implementation manner of the seventeenth aspect correspond to the second aspect and any possible implementation manner of the second aspect respectively, or correspond to the fifth aspect and any possible implementation manner of the fifth aspect respectively. For details, refer to the technical effects of the second aspect and any possible implementation manner of the second aspect, or refer to the technical effects of the fifth aspect and any possible implementation manner of the fifth aspect, which will not be described herein again.

[0153] In an eighteenth aspect, an embodiment of the present application provides a device for transmitting a code stream, the device comprising: a transmitter and at least one storage medium, the at least one storage medium being configured to store the code stream, the code stream being generated according to the second aspect and any one of the implementation manners of the second aspect, or according to the fifth aspect and any one of the implementation manners of the fifth aspect, or according to the seventh aspect, or according to the eighth aspect; and the transmitter being configured to obtain the code stream from the storage medium and transmit the code stream to an end-side device through a transmission medium.

[0154] The eighteenth aspect and any one of the implementation manners of the eighteenth aspect correspond to the second aspect and any one of the implementation manners of the second aspect respectively, or correspond to the fifth aspect and any one of the implementation manners of the fifth aspect respectively. For details, refer to the technical effects of the second aspect and any one of the implementation manners of the second aspect, or refer to the technical effects of the fifth aspect and any one of the implementation manners of the fifth aspect, which will not be described herein again.

[0155] In a nineteenth aspect, an embodiment of the present application provides a system for distributing a code stream, the system comprising: at least one storage medium, configured to store at least one code stream, the at least one code stream being generated according to the second aspect and any one of the implementation manners of the second aspect, or according to the fifth aspect and any one of the implementation manners of the fifth aspect, or according to the seventh aspect, or according to the eighth aspect; and a streaming media device, configured to obtain a target code stream from the at least one storage medium and transmit the target code stream to an end-side device, wherein the streaming media device comprises a content server or a content distribution server.

[0156] The nineteenth aspect and any one of the implementation manners of the nineteenth aspect correspond to the second aspect and any one of the implementation manners of the second aspect respectively, or correspond to the fifth aspect and any one of the implementation manners of the fifth aspect respectively. For details, refer to the technical effects of the second aspect and any one of the implementation manners of the second aspect, or refer to the technical effects of the fifth aspect and any one of the implementation manners of the fifth aspect, which will not be described herein again. BRIEF DESCRIPTION OF DRAWINGS

[0157] Figure 1a An application scenario schematic diagram is exemplarily shown;

[0158] Figure 1b An end-cloud collaborative system schematic diagram is exemplarily shown;

[0159] Figure 2 An encoding process schematic diagram is exemplarily shown;

[0160] Figure 3 Diagram illustrating an exemplary decoding process;

[0161] Figure 4 Diagram illustrating an exemplary encoding process;

[0162] Figure 5 Diagram illustrating an exemplary decoding process;

[0163] Figure 6a Diagram illustrating an exemplary encoding process;

[0164] Figure 6b Diagram illustrating an exemplary model view projection transform;

[0165] Figure 7 Diagram illustrating an exemplary decoding process;

[0166] Figure 8a Diagram illustrating an exemplary encoding process;

[0167] Figure 8b Diagram illustrating an exemplary image;

[0168] Figure 9 Diagram illustrating an exemplary decoding process;

[0169] Figure 10a Diagram illustrating an exemplary encoding process;

[0170] Figure 10b Diagram illustrating an exemplary Render ID;

[0171] Figure 11 Diagram illustrating an exemplary decoding process;

[0172] Figure 12 Diagram illustrating an exemplary encoding process;

[0173] Figure 13 Diagram illustrating an exemplary decoding process;

[0174] Figure 14 Diagram illustrating an exemplary encoding process;

[0175] Figure 15 Diagram illustrating an exemplary decoding process;

[0176] Figure 16a Diagram illustrating an exemplary encoding framework;

[0177] Figure 16b Diagram illustrating an exemplary encoding process;

[0178] Figure 16c An exemplary illustration of a block diagram;

[0179] Figure 16d An exemplary illustration of a CGE information generation process;

[0180] Figure 16e An exemplary illustration of a PU partitioning process;

[0181] Figure 17a An exemplary illustration of a decoding framework;

[0182] Figure 17b An exemplary illustration of a decoding process;

[0183] Figure 18a An exemplary illustration of an encoding framework;

[0184] Figure 18b An exemplary illustration of an encoding process;

[0185] Figure 18c An exemplary illustration of a prediction block selection process;

[0186] Figure 19a An exemplary illustration of a decoding framework;

[0187] Figure 19b An exemplary illustration of a decoding process;

[0188] Figure 20a An exemplary illustration of an encoding framework;

[0189] Figure 20b An exemplary illustration of an encoding process;

[0190] Figure 21a An exemplary illustration of a decoding framework;

[0191] Figure 21b An exemplary illustration of a decoding process;

[0192] Figure 22 An exemplary illustration of a structure of an apparatus. DETAILED DESCRIPTION

[0193] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0194] The term “and / or” in this document is only used to describe associated objects, which means that there can be three relationships, for example, A and / or B can represent three cases: A exists alone, A and B exist together, and B exists alone.

[0195] The terms “first” and “second” and the like in the description and claims of the present application are used to distinguish different objects, and are not used to describe the specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, and are not used to describe the specific order of the target objects.

[0196] In the embodiments of the present application, the words “exemplary” or “for example” are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as “exemplary” or “for example” in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of “exemplary” or “for example” is intended to present relevant concepts in a concrete manner.

[0197] In the description of the embodiments of the present application, unless otherwise specified, the meaning of “a plurality of” is two or more. For example, a plurality of processing units refers to two or more processing units; a plurality of systems refers to two or more systems.

[0198] Figure 1a The application scenario shown is illustrative. Figure 1a The application scenario shown is an end-cloud collaborative rendering scenario. The present application can be applied to various end-cloud collaborative scenarios; for example, cloud games, VR / AR conferences, and the like, and the present application does not limit this. The present application is described taking cloud games as an example.

[0199] Reference Figure 1aIn a possible manner, the terminal device (such as a mobile phone, a PC, a VR glasses, an AR helmet, a tablet computer, and other terminal devices) can send the first rendering parameter to a center server (such as a game server) through an edge server. Then, the center server can perform rendering processing on the three-dimensional scene based on the rendering parameter (the rendering parameter can include only the first rendering parameter uploaded by the terminal device, or include the first rendering parameter uploaded by the terminal device and a second rendering parameter generated by the center server), to obtain a rendered image (such as a game image). Subsequently, the center server encodes the rendered image based on part of the rendering intermediate result (hereinafter referred to as a first rendering intermediate result) generated in the rendering processing process, to obtain a code stream and distribute the code stream to the terminal devices through the edge server. Then, the terminal device can perform rendering processing on the three-dimensional scene based on the rendering parameter (the rendering parameter includes the first rendering parameter and a fifth rendering parameter, the fifth rendering parameter is also generated by the terminal device, and the fifth rendering parameter can include the same parameters as the second rendering parameter), and generate the first rendering intermediate result in the rendering processing process. Then, the terminal device decodes the code stream based on the first rendering intermediate result, to obtain a reconstructed image.

[0200] With reference to Figure 1a In a possible manner, the terminal device (such as a mobile phone, a PC, a VR glasses, an AR helmet, a tablet computer, and other terminal devices) can send the first rendering parameter to a center server through an edge server. Then, the center server can perform rendering processing on the three-dimensional scene based on the rendering parameter (the rendering parameter can include only the first rendering parameter uploaded by the terminal device, or include the first rendering parameter uploaded by the terminal device and a second rendering parameter generated by the center server), to obtain a rendered image. Subsequently, the center server encodes the rendered image based on the first rendering intermediate result generated in the rendering processing process, and encodes a second rendering intermediate result (the second rendering intermediate result is part of the first rendering intermediate result), to obtain a code stream and distribute the code stream to the terminal devices through the edge server. Then, the terminal device can perform rendering processing on the three-dimensional scene based on the rendering parameter (the rendering parameter includes the first rendering parameter and a fifth rendering parameter), and generate a third rendering intermediate result (the third rendering intermediate result is part of the first rendering intermediate result except the second rendering intermediate result) in the rendering processing process. Then, the terminal device generates the first rendering intermediate result based on the second rendering intermediate result and the third rendering intermediate result, and decodes the code stream based on the first rendering intermediate result, to obtain a reconstructed image.

[0201] With reference to Figure 1aIn one possible manner, the first rendering parameters can be sent by a terminal device such as a mobile phone, a PC, a VR glasses, an AR helmet, a tablet computer, and other terminal devices to the center server through an edge server. Then, the center server can perform rendering processing on the three-dimensional scene based on the rendering parameters (the rendering parameters include the first rendering parameters uploaded by the terminal device and the second rendering parameters generated by the center server) to obtain a rendered image. After that, the center server encodes the rendered image based on the first rendering intermediate result generated in the rendering processing process to obtain a code stream and writes the third rendering parameters (the third rendering parameters include all or part of the second rendering parameters) into the code stream. Then, the center server distributes the code stream to each terminal device through the edge server. Subsequently, the terminal device can perform rendering processing on the three-dimensional scene based on the rendering parameters (when the third rendering parameters include all the second rendering parameters, the rendering parameters can include the first rendering parameters and the third rendering parameters parsed from the code stream; when the third rendering parameters include part of the second rendering parameters, the rendering parameters can include the first rendering parameters, the third rendering parameters parsed from the code stream, and the fourth rendering parameters generated by the terminal device, and the fourth rendering parameters are the parameters in the second rendering parameters other than the third rendering parameters) and generate the first rendering intermediate result in the rendering processing process. Then, the terminal device decodes the code stream based on the first rendering intermediate result to obtain a reconstructed image.

[0202] Reference Figure 1aIn one possible manner, the terminal device, such as a mobile phone, a PC, a VR glasses, an AR helmet, a tablet computer, and other terminal devices, can send the first rendering parameter to the center server through the edge server. Then, the center server can perform rendering processing on the three-dimensional scene based on the rendering parameter (including the first rendering parameter uploaded by the terminal device and the second rendering parameter generated by the center server) to obtain a rendered image. Subsequently, the center server encodes the rendered image based on the first rendering intermediate result generated in the rendering processing process, and encodes the second rendering intermediate result to obtain a code stream and write the third rendering parameter into the code stream. Then, the center server distributes the code stream to the terminal devices through the edge server. Subsequently, the terminal device can perform rendering processing on the three-dimensional scene based on the rendering parameter (when the third rendering parameter includes all parameters of the second rendering parameter, the rendering parameter can include the first rendering parameter and the third rendering parameter parsed from the code stream; when the third rendering parameter includes part of the parameters of the second rendering parameter, the rendering parameter can include the first rendering parameter, the third rendering parameter parsed from the code stream, and the fourth rendering parameter generated by the terminal device), and generates the third rendering intermediate result in the rendering processing process. Then, the terminal device generates the first rendering intermediate result based on the second rendering intermediate result and the third rendering intermediate result, and decodes the code stream based on the first rendering intermediate result to obtain a reconstructed image.

[0203] In this way, part or all of the rendering is performed on the terminal device, so that the server can not send the rendering intermediate result or only send part of the rendering intermediate result to the terminal device. Therefore, the application can reduce the interactive delay while ensuring that the code rate overhead of the data stream transmitted by the server to the terminal device is effectively reduced. Moreover, the rendering intermediate result has a strong correlation with the rendered image, and the application encodes the rendered image based on the rendering intermediate result, so that the image reconstruction quality can be ensured. In addition, the data amount of the rendering parameter is much smaller than that of the rendering intermediate result. Therefore, even if the server also sends the rendering parameter to the terminal device, the code rate overhead of the data stream transmitted by the server to the terminal device in the application is also smaller than that of the transmission data stream in the prior art.

[0204] Exemplarily, the rendering parameters can refer to all parameters input to the graphic rendering engine and required by the graphic rendering engine for rendering processing, can include various parameters for rendering, position and color vector of all light sources, position vector of the player or observer, sampling mode of each texture, and information such as position coordinates of each scene object, motion trajectory of a moving object, and bone animation parameters, and the present application does not limit this. The first rendering parameters and the second rendering parameters can constitute the rendering parameters (that is, all parameters input to the graphic rendering engine and required by the graphic rendering engine for rendering processing). It should be noted that the type of the parameters included in the fifth rendering parameters generated by the terminal device is the same as the type of the parameters included in the second rendering parameters generated by the center server, and the precision of the parameters included in the fifth rendering parameters is less than or equal to the precision of the parameters included in the second rendering parameters.

[0205] Exemplarily, the rendering intermediate result can refer to intermediate data generated by the graphic rendering engine for generating the to-be-displayed image / video in the process of generating the to-be-displayed image / video (that is, the rendered image / video). Exemplarily, the rendering intermediate result can include but is not limited to a computer graphics motion vector (CGMV), a rendering intermediate image (the rendering intermediate image refers to an image generated before generating the final rendering image (that is, the rendered image) described above, and the calculation complexity of the rendering intermediate image is lower than that of the rendered image; wherein the rendering intermediate image can be, for example, a rendering intermediate image without indirect light rendering, a rendering intermediate image without mirror reflection processing, a rendering intermediate image without highlight processing, etc.), a position map, a normal map, an albedo map, a specular intensity map, a mesh ID, a material ID (wherein each material map corresponds to a material ID), a rendering ID (each object (or a three-dimensional object model) corresponds to a rendering ID), depth information, etc., and the present application does not limit this. The first rendering intermediate result is part of all rendering intermediate results generated in the rendering processing process, and the second rendering intermediate result and the third rendering intermediate result can constitute the first rendering intermediate result. It should be noted that the type of the intermediate result included in the first rendering intermediate result generated by the terminal device is the same as the type of the intermediate result included in the first rendering intermediate result generated by the center server, and the precision of the intermediate result included in the first rendering intermediate result generated by the terminal device is less than or equal to the precision of the intermediate result included in the first rendering intermediate result generated by the center server.

[0206] Based on the application scenarios of Figure 1a , the application proposes an end-cloud collaborative system, which can be as shown in Figure 1b .

[0207] Figure 1b The end-cloud collaborative system shown is exemplary. Figure 1b The end-cloud collaborative system shown can include a server and a terminal device.

[0208] Referring to Figure 1b , the server can be a central server in Figure 1a , the server can be a single server or a server cluster, and the application does not limit this.

[0209] The server can include a first rendering module, an encoder, and a first communication module. Figure 1b It should be understood that Figure 1b the server of the application can include more or fewer modules than those shown, and the application does not limit this.

[0210] The first rendering module can be used to render a three-dimensional scene based on rendering parameters to obtain a rendered image and a rendering intermediate result.

[0211] The encoder can be used to encode the rendered image based on the first rendering intermediate result, encode the second rendering intermediate result, and write the second rendering parameter into the code stream.

[0212] The first communication module can be used for communication with other electronic devices, for example, the code stream output by the encoder can be sent to the terminal device.

[0213] The terminal device includes but is not limited to: personal computers, computer workstations, smartphones, tablet computers, servers, smart cameras, smart cars or other types of cellular phones, media consumption devices, wearable devices (such as VR / AR headsets, VR glasses), set-top boxes, game consoles, etc.

[0214] Continuing to refer to Figure 1b , the terminal device can include a second communication module, a second rendering module, a decoder, and a display module. Figure 1b It should be understood that Figure 1b the terminal device of the application can include more or fewer modules than those shown, and the application does not limit this.

[0215] The second communication module can be used for communication with other electronic devices, for example, receiving the code stream sent by the server.

[0216] Exemplarily, the second rendering module can be configured to perform rendering processing on the three-dimensional scene based on the locally generated first rendering parameter and the locally generated second rendering parameter to obtain the first rendering intermediate result (at this time, Figure 1b The data flow corresponding to the dashed arrow in FIG. 9 can not exist). Or, the second rendering module can be configured to perform rendering processing on the three-dimensional scene based on the locally generated first rendering parameter and the locally generated second rendering parameter, and generate the third rendering intermediate result based on the second rendering intermediate result sent by the server during the rendering processing. Or, the second rendering module can be configured to perform rendering processing on the three-dimensional scene based on the locally generated first rendering parameter and the server-generated third rendering parameter to obtain the first rendering intermediate result. Or, the second rendering module can be configured to perform rendering processing on the three-dimensional scene based on the locally generated first rendering parameter, the locally generated fourth rendering parameter and the server-generated third rendering parameter to obtain the first rendering intermediate result. Or, the second rendering module can be configured to perform rendering processing on the three-dimensional scene based on the locally generated first rendering parameter and the server-generated third rendering parameter, and generate the third rendering intermediate result based on the second rendering intermediate result sent by the server during the rendering processing. Or, the second rendering module can be configured to perform rendering processing on the three-dimensional scene based on the locally generated first rendering parameter, the locally generated fourth rendering parameter and the server-generated third rendering parameter, and generate the third rendering intermediate result based on the second rendering intermediate result sent by the server during the rendering processing.

[0217] It should be understood that the second rendering module can also generate a rendered image, but the rendered image is not used for subsequent display by the display module.

[0218] Exemplarily, the decoder can be configured to decode the code stream based on the first rendering intermediate result to obtain the reconstructed image.

[0219] Exemplarily, the display module can be configured to display the reconstructed image.

[0220] It should be understood that the video coding standard used by the encoder and the decoder is not limited in the present application, for example, can include but not limited to: H.264 / AVC (Advanced Video Coding, High-level Video Coding), H.265 / HEVC (High Efficiency Video Coding, High Efficiency Video Coding), H.266 / VVC (Versatile Video Coding, Multi-functional Video Coding), AV1 (AOMedia Video 1, AOMedia refers to a video coding developed by the Open Media Video Alliance), etc., and the extension standard of these video coding standards; in addition, it can also include new video coding standards and extension standards generated with the development of video coding technology.

[0221] It should be noted that the encoder involved in the present application is an encoder modified (or optimized) from an existing encoder, and can perform the encoding method involved in the present application, that is, encoding the rendered image based on the first rendering intermediate result. In other words, the encoder involved in the present application can perform the encoding method involved in the present application, and includes all or part of the functions of the existing encoder.

[0222] And the decoder involved in the present application is a decoder modified (or optimized) from an existing decoder, and can perform the decoding method involved in the present application, that is, decoding based on the first rendering intermediate result. In other words, the decoder involved in the present application can perform the decoding method involved in the present application, and includes all or part of the functions of the existing decoder.

[0223] The encoding and decoding processes are described below based on Figure 1a and Figure 1b .

[0224] Figure 2 The encoding process is schematically shown by way of example. In Figure 2 Embodiments, the server does not send the first rendering intermediate result to the terminal device, nor does it send the second rendering parameter to the terminal device. Figure 2 In the embodiments, an example of encoding a frame of rendered image is described.

[0225] S201, the server performs rendering processing on the three-dimensional scene based on the rendering parameter to obtain a rendered image; wherein the rendering parameter includes the first rendering parameter obtained from the terminal device.

[0226] For example, the server can obtain the rendering parameter; then, the rendering parameter can be input to the graphics rendering engine, and the graphics rendering engine can perform rendering processing on the three-dimensional scene based on the rendering parameter to obtain a rendered image. Wherein the graphics rendering engine belongs to Figure 1b The first rendering module in the server.

[0227] Exemplarily, the server can obtain the first rendering parameter from the terminal device (wherein the first rendering parameter is generated by the terminal device). In a possible manner, when the first rendering parameter generated by the terminal device is all parameters required by the graphic rendering engine for rendering processing, the server can determine the first rendering parameter as the rendering parameter; that is, the rendering parameter obtained by the server can only include the first rendering parameter. In a possible manner, when the first rendering parameter generated by the terminal device is part of the parameters required by the graphic rendering engine for rendering processing, the server can further generate a second rendering parameter; then, the first rendering parameter and the second rendering parameter are determined as the rendering parameter. That is, the rendering parameter obtained by the server can include the first rendering parameter and the second rendering parameter. Wherein, the first rendering parameter and the second rendering parameter are specifically described in subsequent embodiments.

[0228] It should be noted that the server can obtain the first rendering parameter from the terminal device according to a preset period, wherein the preset period can be set according to requirements, and the present application does not make any limitation thereon. It should be understood that the first rendering parameter obtained by the server from the terminal device each time can be used for rendering one or more frames of images.

[0229] Exemplarily, the rendering processing process of the graphic rendering engine can be as follows: the image engine can perform geometric transformation, projection transformation, perspective transformation and window clipping on the three-dimensional object model in the three-dimensional scene based on the input rendering parameter, and then generate the rendered image through the obtained material and light information.

[0230] S202, selecting a first rendering intermediate result based on a rendering intermediate result generated in a rendering processing process.

[0231] Exemplarily, the graphic rendering engine in the first rendering module will generate a rendering intermediate result in the rendering processing process (the rendering processing process refers to the time period between the time when the rendering parameter is obtained from the image rendering engine and the time when the rendered image is obtained), and the first rendering intermediate result can be selected from the rendering intermediate result; the first rendering intermediate result can be used for encoding the rendered image subsequently.

[0232] Exemplarily, part of the rendering intermediate result can be selected from the rendering intermediate result as the first rendering intermediate result. Exemplarily, the first rendering intermediate result can be used for any one or more processes in the encoding process, such as blocking, prediction or filtering, etc.; and then, the rendering intermediate result that can be used for any one or more processes in the encoding process can be selected from the rendering intermediate result as the first rendering intermediate result. For example, CGMV, depth information and RenderID are selected as the first rendering intermediate result, and the present application does not make any limitation thereon.

[0233] It should be noted that S201-S202 can be executed by the first rendering module in the first rendering module in S201-S202, S203 can be executed by the encoder in S203. Figure 1b Figure 1b

[0234] S203, the server encodes the rendered image based on the first rendering intermediate result to obtain the encoding data of the residual block and writes the encoding data of the residual block into the bitstream; the first rendering intermediate result acts on at least one processing in the encoding process: blocking, prediction or filtering.

[0235] For example, the encoding process of the encoder can include at least one of the following processes: blocking, prediction or filtering; the first rendering intermediate result can be applied to at least one of the above blocking, prediction or filtering to encode the rendered image.

[0236] For example, in the process of encoding the rendered image, the rendered image can be blocked first to obtain a plurality of to-be-encoded blocks; for a to-be-encoded block, the to-be-encoded block can be predicted based on the filtered reconstructed block to obtain a prediction block; then the residual block between the to-be-encoded block and the prediction block is determined; then the residual block can be encoded and the encoding data of the residual block is written into the bitstream. The specific encoding process is described later

[0237] For example, the residual block can be transformed, quantized and entropy encoded to obtain the encoding data of the residual block.

[0238] It should be understood that the first rendering intermediate result can also act on other processes in the encoding process, such as entropy encoding, which is not limited in the present application.

[0239] In the embodiment, the bitstream includes the encoding data of the residual block, and does not include the encoding data of the first rendering intermediate result. Figure 2

[0240] The decoding process is schematically shown. Figure 3 The decoding process corresponding to the Figure 3 In the embodiment of the first rendering module, the terminal device performs rendering processing based on the locally generated rendering parameter and generates the first rendering intermediate result. Figure 2 In the embodiment, a frame of image is decoded as an example. Figure 3 Figure 3 S301, the terminal device receives the bitstream.

[0241] For example, after the server sends the bitstream, the terminal device can receive the bitstream.

[0242] For example, after the server sends the bitstream, the terminal device can receive the bitstream.

[0243] ​​​S302, the terminal device parses the code stream to obtain a parsing result, the parsing result including a residual block corresponding to the current frame.

[0244] For example, after receiving the code stream, the terminal device can parse the code stream to obtain a parsing result. The parsing process can be as follows: the coding data of the residual block is extracted from the code stream, and then the coding data of the residual block is entropy decoded to obtain entropy decoding data. Then, the entropy decoding data can be inverse quantized to obtain inverse quantization data; and then the inverse quantization data can be inverse transformed to obtain the residual block corresponding to the current frame.

[0245] It should be understood that when the residual block is lossy encoded at the encoding end, the residual block parsed at the decoding end is different from the residual block encoded at the encoding end. When the residual block is lossless encoded at the encoding end, the residual block parsed at the decoding end is the same as the residual block encoded at the encoding end.

[0246] It should be noted that the code stream description information for describing the code stream can also be extracted from the code stream, and at this time, the code stream description information does not need to be entropy decoded, inverse quantized, and inverse transformed. The code stream description information can be used in the subsequent decoding process.

[0247] S303, the terminal device performs rendering processing on the three-dimensional scene based on the rendering parameter corresponding to the current frame, and generates a first rendering intermediate result in the rendering processing. The rendering parameter includes the first rendering parameter generated by the terminal device.

[0248] For example, after the residual block corresponding to the current frame is parsed, the second rendering module (which can be a graphics rendering engine included in the second rendering module) can perform rendering processing on the three-dimensional scene based on the rendering parameter corresponding to the current frame.

[0249] For example, when the first rendering parameter uploaded by the terminal device to the server is all the parameters required for the graphics rendering engine to perform rendering processing, the second rendering module can perform rendering processing on the three-dimensional scene based on the first rendering parameter corresponding to the current frame. When the first rendering parameter uploaded by the terminal device is part of the parameters required for the graphics rendering engine to perform rendering processing, the terminal device can further generate a fifth rendering parameter, and then the second rendering module can perform rendering processing on the three-dimensional scene based on the first rendering parameter corresponding to the current frame and the fifth rendering parameter.

[0250] It should be noted that the type of the parameter included in the fifth rendering parameter generated by the terminal device is the same as the type of the parameter included in the second rendering parameter generated by the central server; the precision of the parameter included in the fifth rendering parameter is less than or equal to the precision of the parameter included in the second rendering parameter.

[0251] It should be noted that the type of the intermediate result contained in the first rendering intermediate result generated by the terminal device is the same as the type of the intermediate result contained in the first rendering intermediate result generated by the server; and the accuracy of the intermediate result contained in the first rendering intermediate result generated by the terminal device is less than or equal to the accuracy of the intermediate result contained in the first rendering intermediate result generated by the server.

[0252] It should be noted that the terminal device can also obtain a rendered image by performing rendering processing by using the graphic rendering engine of the terminal device; since the computing power of the terminal device is lower than the computing power of the server, the image quality of the rendered image obtained by performing rendering processing by using the graphic rendering engine of the terminal device is lower than the image quality of the rendered image obtained by performing rendering processing by using the graphic rendering engine of the server (the image quality can include subjective quality and objective quality (objective quality such as rate-distortion performance, peak signal to noise ratio (PSNR)), and therefore the terminal device can not display the rendered image obtained by performing rendering processing by using the graphic rendering engine of the terminal device, but display the reconstructed image obtained by decoding the bitstream.

[0253] It should be noted that S301-S303 can be performed by a second rendering module in the terminal device, and S304 can be performed by a decoder in the terminal device. Figure 1b

[0254] S304, the terminal device reconstructs based on the first rendering intermediate result and the residual block to obtain a reconstructed image of the current frame; wherein the first rendering intermediate result acts on at least one processing in the reconstruction process: prediction or filtering.

[0255] For example, the decoding process of the decoder can include multiple processes such as parsing, prediction, and filtering, and the first rendering intermediate result can be applied to at least one of the prediction and the filtering to reconstruct the reconstructed image of the current frame.

[0256] For example, in the reconstruction process, the blocking information of the current frame can be determined; then, based on the blocking information and the filtered reconstructed block, the to-be-decoded block in the current frame is predicted to obtain a prediction block; then, the to-be-decoded block and the residual block can be reconstructed based on the prediction block corresponding to the to-be-decoded block to obtain a reconstructed image corresponding to the current frame. The specific decoding process is described later.

[0257] It should be understood that the processing item in the reconstruction process to which the first rendering intermediate result acts in the decoding process corresponds to the processing item in the encoding process to which the first rendering intermediate result acts in the encoding process.

[0258] ​It should be understood that the first rendering intermediate result can also act on other item processing in the decoding process, such as entropy decoding, and the present application does not limit this.

[0259] In this way, the present application places all rendering on the terminal device to execute, and then the server can not send the rendering intermediate result to the terminal device; therefore, the present application can reduce the interactive delay while ensuring that the code rate overhead of the data stream transmitted by the server to the terminal device is effectively reduced. In addition, since the rendering intermediate result has strong correlation with the rendered image, the present application encodes the rendered image based on the rendering intermediate result, which can ensure the image reconstruction quality.

[0260] Figure 4 The encoding process is schematically shown by way of example. In Figure 4 In an embodiment, the server sends part of the first rendering intermediate result (hereinafter referred to as the second rendering intermediate result) to the terminal device, and does not send the second rendering parameter to the terminal device. Figure 4 In an embodiment, a frame of the rendered image is encoded as an example.

[0261] S401, the server performs rendering processing on the three-dimensional scene based on the rendering parameter to obtain a rendered image; wherein the rendering parameter comprises a first rendering parameter obtained from the terminal device.

[0262] S402, the server selects a first rendering intermediate result based on the rendering intermediate result generated in the rendering processing process.

[0263] S403, the server encodes the rendered image based on the first rendering intermediate result to obtain the encoding data of the residual block and writes the encoding data of the residual block into the code stream; the first rendering intermediate result acts on at least one processing in the encoding process: blocking, prediction or filtering.

[0264] S401-S403 can be referred to the description of S201-S203 above, which will not be described here.

[0265] S404, the server encodes the second rendering intermediate result and writes the encoding data of the second rendering intermediate result into the code stream; wherein the second rendering intermediate result is part of the first rendering intermediate result.

[0266] S404 can be performed by the encoder in the above Figure 1b , that is, the encoder encodes the second rendering intermediate result and writes the encoding data of the second rendering intermediate result into the code stream.

[0267] Figure 5 The decoding process is schematically shown by way of example. Figure 5 The decoding process corresponding to Figure 4 is shown. InFigure 5 In one embodiment, the terminal device performs rendering processing based on locally generated rendering parameters and generates a third intermediate rendering result (the third intermediate rendering result and the second intermediate rendering result can form the first intermediate rendering result). Figure 5 In this embodiment, decoding a single frame of image is used as an example for illustration.

[0268] S501, the terminal device receives the bit stream.

[0269] S502, the terminal device parses the bitstream to obtain the parsing result, which includes the residual block corresponding to the current frame and the second rendering intermediate result corresponding to the current frame.

[0270] For example, S502 can be referred to the description of S302 above, and will not be repeated here.

[0271] For example, during the parsing process, the encoded data of the second rendering intermediate result can also be extracted from the bitstream; then, the encoded data of the second rendering intermediate result can be sequentially input into the entropy decoding module, the inverse quantization module, and the inverse transform module to obtain the second rendering intermediate result.

[0272] It should be noted that when the server performs lossless compression on the second intermediate rendering result, the terminal device obtains the same second intermediate rendering result as the one encoded by the server's encoder. However, when the server performs lossy compression on the second intermediate rendering result, the terminal device obtains a different second intermediate rendering result than the one encoded by the server's encoder.

[0273] S503, the terminal device performs rendering processing on the 3D scene based on the rendering parameters corresponding to the current frame, and generates a third rendering intermediate result based on the second rendering intermediate result during the rendering process; wherein, the rendering parameters include the first rendering parameters generated by the terminal device.

[0274] For example, the second rendering module of the terminal device can generate the portion of the first intermediate rendering result excluding the second intermediate rendering result, i.e., the third intermediate rendering result, during the rendering process. For details, please refer to the description in S303 above, which will not be repeated here. Then, the second rendering module can merge the second intermediate rendering result and the third intermediate rendering result to obtain the first intermediate rendering result.

[0275] It should be understood that since the third rendering intermediate result is generated by the terminal device, the type of intermediate result contained in the first rendering intermediate result merged by the terminal device is the same as the type of intermediate result contained in the first rendering intermediate result generated by the server; the precision of the intermediate result contained in the first rendering intermediate result merged by the terminal device is less than or equal to the precision of the intermediate result contained in the first rendering intermediate result generated by the server.

[0276] S504, the terminal device reconstructs based on the first rendering intermediate result and the residual block to obtain a reconstructed image of the current frame; wherein the first rendering intermediate result acts on at least one of the following processes in the reconstruction process: prediction or filtering.

[0277] For example, S504 can refer to the description of S304 described above, which will not be repeated here.

[0278] In this way, the present application places part of the rendering in the terminal device, so that the server can send part of the rendering intermediate result to the terminal device; therefore, the present application can reduce the interaction delay while ensuring that the code rate overhead of the data stream transmitted by the server to the terminal device is effectively reduced. In addition, since the rendering intermediate result has strong correlation with the rendered image, the present application encodes the rendered image based on the rendering intermediate result, which can ensure the image reconstruction quality.

[0279] The following takes the first rendering intermediate result as CGMV and the server sending part of the CGMV to the terminal device as an example to describe the encoding process.

[0280] Figure 6a The encoding process is schematically shown for example. In Figure 6a In the embodiment, the server sends part of the CGMV to the terminal device, and does not send the second rendering parameter to the terminal device; wherein part of the CGMV acts on the prediction in the encoding process. Figure 6a In the embodiment, a frame of rendered image is taken as an example to describe the encoding process.

[0281] S601, the server receives the first rendering parameter sent by the terminal device.

[0282] For example, the first rendering parameter can include camera parameters; wherein the camera parameters can include but are not limited to: view projection matrix, camera dynamic range, focal length, aperture size, depth of field and other parameters.

[0283] S602, the server generates the second rendering parameter.

[0284] For example, the server can load the scene geometry information (such as the vertex coordinates of the triangular patch), material data (such as material map, material generation function) and texture data (such as texture map) and the like pre-existing in the storage (such as hard disk, memory, etc.), and then the second rendering parameter can be obtained.

[0285] S603, the server performs rendering processing on the three-dimensional scene based on the first rendering parameter and the second rendering parameter to obtain a rendered image.

[0286] S604, the server selects the CGMV based on the rendering intermediate result generated in the rendering processing process.

[0287] For example, the server can input the first rendering parameter and the second rendering parameter into a graphic rendering engine (such as V-Ray, Unreal, Unity, etc.), and the graphic rendering engine can perform rendering processing to obtain a rendered image.

[0288] For example, the graphic rendering engine can generate a CGMV and other rendering intermediate results during the rendering processing, and the CGMV can be selected as the first rendering intermediate result. The CGMV can include a CGMV of a static object and a CGMV of a dynamic object.

[0289] For example, the process of generating a static CGMV by the graphic rendering engine can be as follows: for each point (which can include a vertex of a triangular patch and a point on the triangular patch other than the vertex (hereinafter referred to as another point (the other point can be obtained by interpolation based on the vertex))) on the triangular patch (also referred to as a mesh), the projection result of each point at t1 (the current time) under the ModelViewProjection matrix M t1 is calculated (that is, the pixel point of each vertex on the screen), and the projection result is rasterized.

[0290] Figure 6b A schematic diagram of the ModelViewProjection transformation is shown for example.

[0291] Referring to Figure 6b , for example, Figure 6b , the origin of the camera is at (0, 0, 0). +Y represents the positive direction of the Y axis, +X represents the positive direction of the X axis, and +Z represents the positive direction of the Z axis.

[0292] For example, in a three-dimensional scene, the points on the triangular patch between the near plane and the far plane of the view frustum (with the camera origin as the vertex) can be projected onto the screen. In Figure 6b , the points on the gray cube can be projected onto the screen. The parameters of the view frustum (such as the horizontal viewing angle, the vertical viewing angle, and the diagonal viewing angle) can be determined according to the camera parameters in the first rendering parameter. The ModelViewProjection matrix M t1 can be used to determine the pixel points on the screen corresponding to the points on the triangular patch between the near plane and the far plane of the view frustum (with the camera origin as the vertex).

[0293] For example, the process of rasterizing the projection result can be as follows:

[0294] Firstly, preset an array ZBuffer for the pixel points corresponding to the points on the triangular facets between the near plane and the far plane of the frustum (with the camera origin as the vertex); wherein the initial value of the position of each pixel point in the array ZBuffer is set to infinity (i.e. infinitely far from the camera).

[0295] Then, traverse each pixel point [x, y] in the array ZBuffer to calculate the distance z of each pixel point to the camera, that is, the distance of the point corresponding to each pixel point on the triangular facet to the camera. When the pixel point corresponds to a vertex of the triangular facet, the corresponding distance can be directly calculated; when the pixel point corresponds to other points on the triangular facet, the corresponding distance can be obtained by interpolating the distances of the vertex corresponding pixel points to the camera. If z is less than the value in ZBuffer[x, y], update the value of ZBuffer[x, y] to the distance z of the point.

[0296] Subsequently, based on the far plane distance value z far and the near plane distance value z near in the camera parameters, normalize the distance values corresponding to each pixel point in the array ZBuffer to obtain depth values D in the range [0, 1], and all the depth values D form a depth map. The normalization method is:

[0297]

[0298] Then, according to the view projection matrix M t2 of the t2 moment (any reconstruction image corresponds to a rendering moment; wherein the graphics rendering engine does not necessarily render according to the image arrangement order in the video data, so the encoder also does not necessarily encode according to the image arrangement order in the video data; further, the reconstruction image can be a reconstruction image of an encoded image located before the to-be-rendered image (i.e. the image to be rendered at the t1 moment) in the video data, or a reconstruction image of an encoded image located after the to-be-rendered image) and the view projection matrix M t1 of the current moment, calculate the CGMV of the static object between the t1 moment and the t2 moment, denoted as CGMV1:

[0299] CGMV1=D*(M t1 -M t2 )

[0300] It should be understood that the CGMV of the static object between the t1 moment and N t2 moments can be generated, and then N CGMV1s can be obtained, each of which can correspond to a t2 moment. Wherein N is a positive integer, for example, N is a positive integer less than 16, which is not limited in the present application.

[0301] For example, the process of generating dynamic CGMV by the graphic rendering engine can be as follows:

[0302] For example, a velocity buffer is maintained for the mesh that is in motion (i.e. the triangle patches of the dynamic object), in which the coordinates of each vertex on each mesh that is in motion at t1 are recorded as (x1, y1, z1), and the coordinates of each vertex on each mesh that is in motion at t2 are recorded as (x2, y2, z2). Further,

[0303] First, the motion vector P = (x1, y1, z1) - (x2, y2, z2) of each vertex on each mesh that is in motion at t1 and t2 can be calculated.

[0304] Next, P is multiplied by the view projection matrix at t1 to obtain the CGMV of each vertex on each mesh that is in motion corresponding to the pixel point, denoted as CGMV2, CGMV2 = P * M t1 .

[0305] Finally, rasterization is performed on CGMV2; for details, refer to the description above, which will not be repeated here. In addition, interpolation can be performed on other points on the triangle patch to obtain CGMV2 for each pixel point.

[0306] It should be understood that the CGMV of the dynamic object between t1 and N t2 can be generated, and N CGMV2 can be obtained, each of which can correspond to a t2.

[0307] S605, the server encodes the rendered image based on the CGMV to obtain the encoding data of the residual block and writes the encoding data of the residual block into the bitstream; wherein the CGMV acts on one of the following processes in the encoding process: prediction.

[0308] For example, the server can combine the CGMV of the dynamic object and the CGMV of the static object according to the pixel point to obtain the complete CGMV (i.e. the first rendering intermediate result). For example, the CGMV1 of the static object between t1 and the i-th (i is a positive integer between 1 and N, inclusive) t2 and the CGMV2 of the dynamic object between t1 and the i-th t2 can be combined to obtain the complete CGMV (i.e. the i-th complete CGMV) between t1 and the i-th t2.

[0309] Exemplarily, the CGMV can be applied to prediction in an encoding process to encode the rendered image. Exemplarily, for a to-be-encoded block in the rendered image, the to-be-encoded block can be predicted based on the CGMV to obtain a predicted block; then, a residual block between the to-be-encoded block and the predicted block is determined; and then, the residual block can be encoded, and the encoding data of the residual block is written into a bitstream. The specific encoding process is described in subsequent embodiments.

[0310] In S606, the server encodes the first CGMV, and writes the encoding data of the first CGMV into a bitstream.

[0311] Exemplarily, after the server generates the complete CGMV, the server can encode part of the CGMV. It should be understood that when the predicted block used to calculate the residual block is the i th complete CGMV, the server can encode part of the i th complete CGMV. The part of the CGMV can be referred to as a first CGMV, that is, a second rendering intermediate result.

[0312] In a possible manner, the first CGMV is a CGMV of a static object, and then the server can send the encoded CGMV of the static object to the terminal device. In this way, the terminal device does not need to calculate the CGMV of the static object, and the computing power of the terminal device is saved.

[0313] In a possible manner, the first CGMV is a CGMV of a dynamic object, and then the server can send the encoded CGMV of the dynamic object to the terminal device. In this way, the terminal device does not need to calculate the CGMV of the dynamic object, and the computing power of the terminal device is saved. In addition, compared with calculating the CGMV of the static object, the calculation amount of calculating the CGMV of the dynamic object is larger, and therefore, sending the CGMV of the dynamic object after encoding to the terminal device can save more computing power of the terminal device.

[0314] Exemplarily, the CGMV of the dynamic object can include a CGMV of a rigid dynamic object and / or a CGMV of a non-rigid dynamic object.

[0315] In a possible manner, the first CGMV can include a CGMV of a rigid dynamic object and a CGMV of a non-rigid dynamic object.

[0316] In a possible manner, the first CGMV can include a CGMV of a rigid dynamic object, and then the server can send the encoded CGMV of the rigid dynamic object to the terminal device. In this way, compared with encoding the CGMV of the rigid dynamic object and the CGMV of the non-rigid dynamic object, the code rate can be reduced.

[0317] In one possible approach, the first CGMV can include the CGMV of a non-rigid dynamic object, and the server can then send the encoded CGMV of the non-rigid dynamic object to the terminal device. This reduces the bitrate compared to encoding the CGMV of rigid dynamic objects. Furthermore, since non-rigid dynamic objects have a large number of triangular faces, the computational load for their CGMVs is significant; therefore, encoding the CGMV of non-rigid dynamic objects saves more computing power on the terminal device compared to encoding the CGMV of rigid dynamic objects.

[0318] For example, in Figure 6a In some embodiments, the bitstream description information may further include a first indicator and / or a second indicator. The first indicator indicates whether the bitstream includes encoded data of the second rendering intermediate result; the second indicator indicates the type and subtype of the second rendering intermediate result.

[0319] For example, the type of CGMV can be a motion vector class, which can include subtypes A1, A2, and A3; wherein, the subtype corresponding to the CGMV of a static object is subtype A1, the subtype corresponding to the CGMV of a rigid dynamic object is subtype A2, and the subtype corresponding to the CGMV of a non-rigid dynamic object is subtype A3. Figure 6a In one embodiment, the second indicator can be used to indicate the type and subtype of the first CGMV; thus, the terminal device can know what type of second CGMV is generated (the second CGMV is the part of the CGMV other than the first CGMV).

[0320] For example, if the motion vector class is represented by V, subtype A1 is represented by 1, subtype A2 by 2, and subtype A3 by 3, then when the first CGMV is the CGMV of a non-rigid dynamic object, the second indicator can be V3.

[0321] For example, when the first indicator is a first preset information (such as 1), it means that the bitstream contains encoded data of the second rendering intermediate result; when the first indicator is a second preset information (such as 0), it means that the bitstream does not contain encoded data of the second rendering intermediate result.

[0322] The following example uses the bitstream description information, which includes a first indicator and a second indicator, to illustrate this. Figure 6a The decoding process corresponding to the encoding process will be explained.

[0323] Figure 7 This is a schematic diagram illustrating the decoding process as an example. Figure 7 What is shown is with Figure 6a The corresponding decoding process. In Figure 7In an embodiment, the terminal device performs rendering processing based on the locally generated rendering parameter, and generates another part of the CGMV. Figure 7 In an embodiment, a frame of image is taken as an example for description.

[0324] In S701, the terminal device receives a code stream.

[0325] In S702, the terminal device parses the code stream to obtain a parsing result, the parsing result including a residual block corresponding to a current frame and a first CGMV corresponding to the current frame.

[0326] For example, S701-S702 can refer to the description of S501-S502, which will not be repeated here.

[0327] In S703, the terminal device generates a first rendering parameter and a fifth rendering parameter.

[0328] For example, the first rendering parameter can include camera parameters, which can include but are not limited to: view projection matrix, camera dynamic range, focal length, aperture size, and depth of field, etc.

[0329] For example, the terminal device can load scene geometry information (such as vertex coordinates of triangular patches), material data (such as material maps, material generation functions), and texture data (such as texture maps) pre-stored in the memory (such as hard disk, memory, etc.), and then obtain the fifth rendering parameter.

[0330] In S704, the terminal device performs rendering processing on a three-dimensional scene based on the first rendering parameter and the fifth rendering parameter, and generates a second CGMV based on the first CGMV during the rendering processing.

[0331] For example, the terminal device can input the first rendering parameter and the fifth rendering parameter to a graphics rendering engine (such as V-Ray, Unreal, Unity, etc.), and perform rendering processing on the three-dimensional scene by the graphics rendering engine; and during the rendering processing, generate the second CGMV based on the first CGMV.

[0332] For example, the parsing result can further include a first indication identifier and a second indication identifier. When the terminal device determines that the code stream includes encoding data of the second rendering intermediate result according to the first indication identifier, the terminal device can determine the type and the sub-type of the second rendering intermediate result according to the second indication identifier. Then, during the rendering processing, generate a third rendering intermediate result of another sub-type based on the type and the sub-type of the second rendering intermediate result; wherein the another sub-type is a sub-type in the sub-types of the type of the second rendering intermediate result, except for the sub-type corresponding to the second rendering intermediate result.

[0333] For example, in the embodiment, the terminal device can determine the type of the second rendering intermediate result according to the first indication identifier, and determine the sub-type of the second rendering intermediate result according to the second indication identifier. Figure 7In the embodiment, when the terminal device determines that the type of the first CGMV is a motion vector class and the subtype is subtype A1 according to the second indication identifier, that is, the first CGMV sent by the server is the CGMV of a static object; then the graphics rendering engine of the terminal device does not need to generate the CGMV of the static object during the rendering process, but only generates the CGMV of the dynamic object (at this time, other subtypes include subtype A2 and subtype A3); that is, the second CGMV is the CGMV of the dynamic object.

[0334] For example, in Figure 7 In the embodiment, when the terminal device determines that the type of the first CGMV is a motion vector class and the subtypes are subtype A2 and subtype A3 according to the second indication identifier, that is, when the first CGMV sent by the server is a CGMV of a rigid dynamic object and a CGMV of a non-rigid dynamic object, the graphics rendering engine of the terminal device does not need to generate CGMVs of rigid dynamic objects and non-rigid dynamic objects during the rendering process, but only generates CGMVs of static objects (at this time, other subtypes include subtype A1); that is, the second CGMV is a CGMV of a static object.

[0335] For example, in Figure 7 In the embodiment, when the terminal device determines that the type of the first CGMV is a motion vector class and the subtype is subtype A2 according to the second indication identifier, that is, when the first CGMV sent by the server is a CGMV of a rigid dynamic object, the graphics rendering engine of the terminal device does not need to generate a CGMV of a rigid dynamic object during the rendering process, but only generates a CGMV of a static object and a CGMV of a non-rigid dynamic object (at this time, other subtypes include subtypes A1 and A3); that is, the second CGMV is a CGMV of a static object and a CGMV of a non-rigid dynamic object.

[0336] For example, in Figure 7 In the embodiment, when the terminal device determines that the type of the first CGMV is a motion vector class and the subtype is subtype A3 according to the second indication identifier, that is, when the first CGMV sent by the server is a CGMV of a non-rigid dynamic object, the graphics rendering engine of the terminal device does not need to generate a CGMV of a non-rigid dynamic object during the rendering process, but only generates a CGMV of a static object and a CGMV of a rigid dynamic object (at this time, other subtypes include subtypes A1 and A2); that is, the second CGMV is a CGMV of a static object and a CGMV of a rigid dynamic object.

[0337] It should be understood that the process of a terminal device generating a CGMV of a dynamic object / a CGMV of a static object can be referred to the description above, and will not be repeated here.

[0338] S705, the terminal device reconstructs the image of the current frame based on the first CGMV, the second CGMV, and the residual block; wherein, the first intermediate rendering result is applied to the following processing in the reconstruction process: prediction.

[0339] For example, the terminal device can merge the first CGMV and the second CGMV; then, based on the merged CGMV (that is, the complete CGMV), it can be used to perform predictions in the reconstruction process to obtain the reconstructed image of the current frame.

[0340] For example, during the reconstruction process, prediction can be performed on the block to be decoded in the current frame based on CGMV to obtain the predicted block; then, reconstruction can be performed based on the residual block corresponding to the block to be decoded and the predicted block to obtain the reconstructed image corresponding to the current frame. The specific reconstruction process will be explained later and will not be repeated here.

[0341] Because the CGMV generated by the server is more accurate than that generated by the terminal device, the CGMV obtained by the terminal device is more accurate, thereby improving prediction accuracy and ultimately enhancing the image quality of the reconstructed image. Furthermore, the terminal device only needs to generate a portion of the CGMV, saving computing power and improving decoding efficiency. In addition, under the same quality conditions, compared to existing technologies where the server sends all the CGMVs, the server in this application only sends a portion of the CGMVs, which also reduces the bitrate overhead of the data stream transmitted from the server to the terminal device.

[0342] The following explanation uses the first intermediate rendering result as depth information, with the server sending part of the depth information to the terminal device, as an example to illustrate the encoding and decoding process. For instance, the depth information is the depth map.

[0343] Figure 8a This is a schematic diagram illustrating an exemplary encoding process. Figure 8a In one embodiment, the server sends a partial region image from the depth map to the terminal device, but does not send the second rendering parameters to the terminal device. Figure 8a In this embodiment, the example of encoding a rendered image frame is used for illustration.

[0344] S801, the server receives the first rendering parameters sent by the terminal device.

[0345] S802, the server generates the second rendering parameters.

[0346] For example, S801 to S802 can be referred to the description of S601 to S602 above, and will not be repeated here.

[0347] S803, the server renders the 3D scene based on the first rendering parameters and the second rendering parameters to obtain the rendered image.

[0348] S804, the server selects the depth map based on the rendering intermediate result generated in the rendering process.

[0349] For example, the depth map can include depth information of each pixel in the rendered image; for details, please refer to Figure 8b , wherein, Figure 8b (1) is the rendered image, Figure 8b (2) is the depth map.

[0350] S805, the server encodes the rendered image based on the depth map to obtain the encoding data of the residual block and writes the encoding data of the residual block into the bitstream; wherein the depth map acts on the following one processing in the encoding process: blocking.

[0351] For example, the depth map can be applied to the blocking in the encoding process to encode the rendered image. For example, the rendered image can be blocked based on the depth map to obtain a to-be-encoded block; then, the to-be-encoded block can be predicted to obtain a predicted block; then, the residual block between the to-be-encoded block and the predicted block is determined; then, the residual block can be encoded, and the encoding data of the residual block is written into the bitstream. The specific encoding process is described in subsequent embodiments.

[0352] S806, the server encodes the first area image in the depth map and writes the encoding data of the first area image into the bitstream.

[0353] For example, the server can encode the first area image in the depth map after generating the complete depth map.

[0354] In one possible way, the first area image can refer to the image of the area corresponding to the four corners in the depth map. The size of the image of each corner corresponding area can be set according to the needs, and the present application does not limit this.

[0355] In one possible way, the first area image can refer to the image of the area in the depth map except the area corresponding to the four corners.

[0356] It should be understood that the first area image can also be the image of other areas in the depth map, and the present application does not limit this.

[0357] For example, in the embodiment of Figure 8a , the above bitstream description information can further include a first indication identifier and / or a second indication identifier. The second indication identifier can be used to indicate the type and subtype of the second rendering intermediate result; the first indication identifier can be used to indicate whether the encoding data of the second rendering intermediate result is contained in the bitstream.

[0358] For example, the depth map is of type first image class, which may include subtype B1 and subtype B2; for instance, the subtype corresponding to the regions at the four corners of the depth map is subtype B1, and the subtype corresponding to the regions other than the regions at the four corners of the depth map is subtype B2. Figure 8a In one embodiment, the second indicator can be used to indicate the type and subtype of the first region image; thus, the terminal device can know what kind of region image is generated, namely the second region image (the second region image is the image in the depth map other than the first region image).

[0359] For example, if the first image class is represented by M, subtype B1 by 1, and subtype B2 by 2, then when the first region image is the image of the region corresponding to the four corners of the depth map, the second indicator can be M1.

[0360] The following example uses the bitstream description information, which includes a first indicator and a second indicator, to illustrate this. Figure 8a The decoding process corresponding to the encoding process will be explained.

[0361] Figure 9 This is a schematic diagram illustrating the decoding process as an example. Figure 9 What is shown is with Figure 8a The corresponding decoding process. In Figure 9 In one embodiment, the terminal device performs rendering processing based on locally generated rendering parameters and generates images of a portion of the depth map. Figure 9 In this embodiment, decoding a single frame of image is used as an example for illustration.

[0362] S901, the terminal device receives the code stream.

[0363] S902, the terminal device parses the bitstream to obtain the parsing result, which includes the residual block corresponding to the current frame and the first region image corresponding to the current frame.

[0364] For example, S901 to S902 can be referred to the description of S501 to S502 above, and will not be repeated here.

[0365] S903, the terminal device generates the first rendering parameters and the fifth rendering parameters.

[0366] For example, S903 can be referred to the description of S703 above, and will not be repeated here.

[0367] S904, the terminal device performs rendering processing based on the first rendering parameters and the fifth rendering parameters, and generates a second region image based on the first region image during the rendering process.

[0368] Exemplarily, the terminal device can input the first rendering parameter and the fifth rendering parameter to a graphic rendering engine (such as V-Ray, Unreal, Unity, etc.), and render the three-dimensional scene by the graphic rendering engine; and in the rendering process, the second region image is generated based on the first region image.

[0369] Exemplarily, the analysis result can further include a first indication identifier and a second indication identifier. When the terminal device determines, according to the first indication identifier, that the code stream includes the encoding data of the second rendering intermediate result, the type and the sub-type of the second rendering intermediate result can be determined according to the second indication identifier. Then, in the rendering process, the third rendering intermediate result of other sub-type is generated based on the type and the sub-type of the second rendering intermediate result; wherein the other sub-type is a sub-type in the sub-types of the type of the second rendering intermediate result, except for the sub-type corresponding to the second rendering intermediate result.

[0370] Exemplarily, in the embodiment of Figure 9 , when the terminal device determines, according to the second indication identifier, that the type of the first region image is the first image class and the sub-type is the sub-type B1, that is, the first region image sent by the server is the image of the four corner corresponding regions in the depth map; then in the rendering process of the graphic rendering engine of the terminal device, the image of the four corner corresponding regions does not need to be generated, and only the image of the region except for the four corner corresponding regions (at this time, the other sub-type is the sub-type B2) can be generated; that is, the second region image is the image of the region except for the four corner corresponding regions.

[0371] Exemplarily, in the embodiment of Figure 9 , when the terminal device determines, according to the second indication identifier, that the type of the first region image is the first image class and the sub-type is the sub-type B2, that is, the first region image sent by the server is the image of the region except for the four corner corresponding regions in the depth map; then in the rendering process of the graphic rendering engine of the terminal device, the image of the region except for the four corner corresponding regions does not need to be generated, and only the image of the four corner corresponding regions (at this time, the other sub-type is the sub-type B1) can be generated; that is, the second region image is the image of the four corner corresponding regions.

[0372] S905, the terminal device reconstructs based on the first region image, the second region image and the residual block to obtain the reconstructed image of the current frame; wherein the first rendering intermediate result acts on the following one processing in the reconstruction process: prediction.

[0373] Exemplarily, the first region image and the second region image can be merged according to pixels to obtain the depth map; and then the depth map and the residual block are used for reconstruction to obtain the reconstructed image of the current frame.

[0374] For example, the prediction in the reconstruction process needs to use the block information of the current frame, and therefore the block information can be determined based on the depth map; then, a to-be-decoded block in the current frame is predicted based on the block information and the filtered reconstructed block to obtain a prediction block; and then, the to-be-decoded block can be reconstructed based on the prediction block corresponding to the to-be-decoded block and the residual block to obtain a reconstructed image corresponding to the current frame. The specific decoding process is described below.

[0375] Since the depth map generated by the server is more accurate than the depth map generated by the terminal device, the terminal device can obtain a more accurate depth map, thereby improving the accuracy of the block information and the image quality of the reconstructed image. In addition, the terminal device only needs to generate images of part of the depth map, which can save the computing power of the terminal device and improve the decoding efficiency. Furthermore, compared with the prior art in which the server sends all CGMVs, the server in the present application only sends images of part of the depth map, which can also reduce the code rate overhead of the data stream transmitted by the server to the terminal device.

[0376] The encoding and decoding processes are described below by taking the first rendering intermediate result as a Render ID and the server sending part of the Render ID to the terminal device as an example. The Render ID refers to a two-dimensional image, and the Render ID can be referred to as a Render ID image.

[0377] Figure 10a The encoding process is schematically shown by way of example. In Figure 10a In the embodiment, the server sends part of the images in the Render ID image to the terminal device, and does not send the second rendering parameter to the terminal device. Figure 10a In the embodiment, the encoding of a frame of rendered image is described by way of example.

[0378] S1001, the server receives the first rendering parameter sent by the terminal device.

[0379] S1002, the server generates a second rendering parameter.

[0380] For example, S1001-S1002 can refer to the description of S601-S602 described above, and will not be described here.

[0381] S1003, the server performs rendering processing on the three-dimensional scene based on the first rendering parameter and the second rendering parameter to obtain a rendered image.

[0382] S1004, the server selects a Render ID image based on the rendering intermediate result generated in the rendering processing.

[0383] For example, the process of generating the Render ID map by the graphic rendering engine can be as follows: in the three-dimensional scene, the points on the triangular patches between the near plane and the far plane of the view frustum (with the camera origin as the vertex) can be projected onto the screen. For example, in the process of projecting the triangular patches onto the screen, the object ID corresponding to each vertex of the triangular patches that can be observed by the camera (i.e., located between the near plane and the far plane of the view frustum) can be queried; then, for each vertex, the pre-stored color table is queried based on the object ID corresponding to the vertex to determine the color of the vertex, i.e., the color of the pixel point on the screen corresponding to the vertex. The color of the pixel point on the screen corresponding to other points inside the triangular patch can be determined by interpolation using the colors of the vertices of the triangular patch; in this way, the Render ID map can be obtained.

[0384] Figure 10b An exemplary Render ID map is shown. In the exemplary Render ID map, different gray areas correspond to different colors. Figure 10b

[0385] In S1005, the server encodes the rendered image based on the Render ID to obtain the encoding data of the residual block and writes the encoding data of the residual block into the bitstream; wherein the Render ID is applied to one of the following processes in the encoding process: filtering.

[0386] For example, the Render ID map can be applied to filtering in the encoding process to filter the reconstructed block; subsequently, the filtered reconstructed block can be used as a reference to predict the to-be-encoded block in the rendered image to obtain a prediction block; then, the residual block between the to-be-encoded block and the prediction block can be determined; subsequently, the residual block can be encoded, and the encoding data of the residual block can be written into the bitstream; the specific encoding process is described in subsequent embodiments.

[0387] In S1006, the server encodes the third area image in the Render ID and writes the encoding data of the third area image into the bitstream.

[0388] For example, after generating the complete Render ID map, the server can encode the third area image in the Render ID map.

[0389] In one possible manner, the third area image can refer to the image of the area corresponding to each corner in the Render ID map. The image size of each corner corresponding area can be set as required, and the present application does not limit this.

[0390] In one possible manner, the third area image can refer to the image of the area in the Render ID map except the area corresponding to each corner.

[0391] ​It should be understood that the third region image can also be an image of other regions in the Render ID image, and this application does not impose any restrictions on this.

[0392] For example, in Figure 10a In some embodiments, the bitstream description information may further include a first indicator and / or a second indicator. The second indicator may be used to indicate the type and subtype of the second rendering intermediate result; the first indicator may be used to indicate whether the bitstream contains encoded data of the second rendering intermediate result.

[0393] For example, the type of the Render ID image can be a second image class, which can include subtype B3 and subtype B4; for instance, the subtype corresponding to the areas at the four corners of the Render ID image is subtype B3, and the subtype corresponding to the areas other than the areas at the four corners of the Render ID image is subtype B4. Figure 10a In one embodiment, the second indicator can be used to indicate the type and subtype of the third region image; thus, the terminal device can know which regions of the image are generated, i.e., the fourth region image (the fourth region image is the image in the Render ID image other than the third region image).

[0394] For example, if the second image class is represented by K, subtype B3 by 1, and subtype B4 by 2, then when the third region image is the image of the four corner regions in the Render ID image, the second indicator can be K1.

[0395] The following example uses the bitstream description information, which includes a first indicator and a second indicator, to illustrate this. Figure 10a The decoding process corresponding to the encoding process will be explained.

[0396] Figure 11 This is a schematic diagram illustrating the decoding process as an example. Figure 11 What is shown is with Figure 10a The corresponding decoding process. In Figure 11 In one embodiment, the terminal device performs rendering processing based on locally generated rendering parameters and generates a partial area image in the Render ID map. Figure 11 In this embodiment, decoding a single frame of image is used as an example for illustration.

[0397] S1101, the terminal device receives the bit stream.

[0398] S1102, the terminal device parses the bitstream to obtain the parsing result, which includes the residual block corresponding to the current frame and the third region image corresponding to the current frame.

[0399] For example, S1101-S1102 can refer to the description of S501-S502, which will not be repeated here.

[0400] S1103, the terminal device generates the first rendering parameter and the fifth rendering parameter.

[0401] For example, S1103 can refer to the description of S703, which will not be repeated here.

[0402] S1104, the terminal device performs rendering processing based on the first rendering parameter and the fifth rendering parameter, and generates a fourth region image based on the third region image during the rendering processing.

[0403] For example, the terminal device can input the first rendering parameter and the fifth rendering parameter to a graphics rendering engine (such as V-Ray, Unreal, Unity, etc.), and perform rendering processing on the three-dimensional scene by the graphics rendering engine; and during the rendering processing, generate a fourth region image based on the third region image.

[0404] For example, the analysis result can also include a first indication identifier and a second indication identifier. When the terminal device determines that the code stream includes the second rendering intermediate result according to the first indication identifier, it can determine the type and subtype of the second rendering intermediate result according to the second indication identifier. Then, during the rendering processing, generate a third rendering intermediate result of other subtypes based on the type and subtype of the second rendering intermediate result; wherein the other subtypes are subtypes in the subtypes contained by the type of the second rendering intermediate result, except for the subtype corresponding to the second rendering intermediate result.

[0405] For example, in the embodiment of Figure 11 When the terminal device determines that the type of the third region image is the second image class and the subtype is subtype B3 according to the second indication identifier, that is, the third region image sent by the server is the image of the four corner corresponding regions in the Render ID image; then the graphics rendering engine of the terminal device does not need to generate the image of the four corner corresponding regions during the rendering processing, and only generates the image of the region except the four corner corresponding regions (at this time, the other subtypes are subtype B4); that is, the fourth region image is the image of the region except the four corner corresponding regions.

[0406] For example, in the embodiment of Figure 11In the embodiment of the first aspect, when the terminal device determines, according to the second indication identifier, that the type of the third region image is the second image category and the sub-type is the sub-type B4, that is, the third region image sent by the server is an image of a region other than the four corner regions in the Render ID image, the terminal device does not need to generate an image of the region other than the four corner regions in the rendering process of the graphic rendering engine of the terminal device, and only needs to generate an image of the four corner regions (at this time, the other sub-type is the sub-type B3); that is, the fourth region image is an image of the four corner regions.

[0407] In the embodiment of the first aspect, the terminal device reconstructs, based on the third region image, the fourth region image, and the residual block, to obtain a reconstructed image of the current frame; and the first rendering intermediate result is used in filtering in the reconstruction process.

[0408] In an example, the third region image and the fourth region image can be combined according to pixels to obtain a Render ID image. In an example, the Render ID image can be used in filtering in the decoding process to filter the reconstructed block; subsequently, the filtered reconstructed block can be used as a reference to predict a to-be-decoded block of the current frame to obtain a prediction block; and then, the prediction block and the residual block are used to reconstruct to obtain a reconstructed image of the current frame. The specific encoding process is described in subsequent embodiments.

[0409] Since the Render ID image generated by the server is more accurate than the Render ID image generated by the terminal device, the terminal device can obtain a more accurate Render ID image, thereby improving the quality of the reference reconstructed block and the image quality of the reconstructed image. In addition, the terminal device only needs to generate an image of a part of the region in the Render ID image, which can save the computing power of the terminal device and improve the decoding efficiency. Furthermore, in the case of the same quality, compared with the prior art in which the server sends all CGMVs, the server in the present application only sends images of a part of the region in the Render ID image, which can also reduce the code rate overhead of the data stream transmitted by the server to the terminal device.

[0410] It should be noted that when the code stream description parameter of the code stream does not include the first indication identifier, Figure 7 , Figure 9 and Figure 11 In the embodiments of the first aspect, the terminal device can determine, according to the pre-agreed information, whether the code stream includes the encoding data of the second rendering intermediate result. When the code stream description parameter of the code stream does not include the second indication identifier, Figure 7 , Figure 9 and Figure 11 In the embodiments of the first aspect, the terminal device can determine, according to the pre-agreed information, the type and sub-type of the second rendering intermediate result in the parsing result.

[0411] It should be understood that when the first rendering intermediate result is other information, the above-mentioned coding manner can also be referred to; and whether to send all / part of the other information to the terminal device can be determined according to the data amount of the other information, and the present application does not limit this.

[0412] For example, when the first rendering parameter generated by the terminal device is part of the parameter required by the graphic rendering engine for rendering processing, the server can generate the second rendering parameter. Since the rendering parameter generated by the server is more accurate than the rendering parameter generated by the terminal device, the server can send part or all of the second rendering parameter to the terminal device, so that the first rendering intermediate result generated by the terminal device is more accurate, thereby improving the image quality of the image decoded based on the first rendering intermediate result.

[0413] Figure 12 An example encoding process schematic diagram is shown. In Figure 12 In the embodiment, the server does not send the first rendering intermediate result to the terminal device, but sends the third rendering parameter (part or all of the second rendering parameter) to the terminal device. Figure 12 In the embodiment, an example of encoding a frame of rendered image is described.

[0414] S1201, the server receives the first rendering parameter sent by the terminal device.

[0415] For example, the first rendering parameter can include camera parameters. The camera parameters can include but are not limited to view projection matrix, camera dynamic range, focal length, aperture size, depth of field, etc.

[0416] S1202, the server generates the second rendering parameter.

[0417] For example, the server can load the scene geometry information (such as the vertex coordinates of the triangular patch), material data (such as material map, material generation function), and texture data (such as texture map), motion information of dynamic objects, etc. pre-existing in the memory (such as hard disk, memory, etc.), and then the second rendering parameter can be obtained.

[0418] For example, the motion information of the dynamic object can include the motion information (such as motion trajectory) of the rigid dynamic object and / or the motion information (such as motion trajectory) of the non-rigid dynamic object.

[0419] S1203, the server performs rendering processing on the three-dimensional scene based on the first rendering parameter and the second rendering parameter to obtain a rendered image.

[0420] S1204, the server selects the first rendering intermediate result based on the rendering intermediate result generated in the rendering processing process.

[0421] Exemplarily, in the embodiments of the first rendering intermediate result can be CGMV, depth information or RenderID. Figure 12

[0422] S1205, the server encodes the rendered image based on the first rendering intermediate result to obtain the encoding data of the residual block and writes the encoding data of the residual block into the bitstream; the first rendering intermediate result acts on at least one processing in the encoding process: blocking, prediction or filtering.

[0423] Exemplarily, S1203-S1205 can refer to the description of the above embodiments, which will not be repeated here.

[0424] S1206, the server writes the third rendering parameter into the bitstream.

[0425] Exemplarily, the third rendering parameter can be part of the second rendering parameter, or all of the second rendering parameter; the present application does not limit this. Compared with writing all the second rendering parameter into the bitstream, writing part of the second rendering parameter into the bitstream can further reduce the code rate overhead of the data stream transmitted by the server to the terminal device.

[0426] Exemplarily, since the data amount of the third rendering parameter is small, the server can encode the third rendering parameter and write the encoding data of the third rendering parameter into the bitstream, and then send it to the server; the server can also not encode the third rendering parameter, but directly write the third rendering parameter into the bitstream; the present application does not limit this.

[0427] In one possible way, the third rendering parameter can include the motion information of the rigid moving object and the motion information of the non-rigid dynamic object, i.e., the motion information of the rigid moving object and the motion information of the non-rigid dynamic object are written into the bitstream.

[0428] In one possible way, the third rendering parameter can include the motion information of the rigid moving object, i.e., the motion information of the rigid moving object is written into the bitstream. In this way, compared with writing the motion information of the rigid moving object and the motion information of the non-rigid dynamic object into the bitstream, the code rate overhead of the data stream transmitted by the server to the terminal device can be further reduced.

[0429] In one possible way, the third rendering parameter can include the motion information of the non-rigid moving object, i.e., the motion information of the non-rigid dynamic object is written into the bitstream. In this way, compared with writing the motion information of the rigid moving object and the motion information of the non-rigid dynamic object into the bitstream, the code rate overhead of the data stream transmitted by the server to the terminal device can be further reduced.

[0430] ​In this way, the amount of data for the third rendering parameter is much smaller than the amount of data for the second rendering intermediate result. Compared with transmitting the second rendering intermediate result, transmitting the third rendering parameter can further reduce the bit rate.

[0431] For example, in Figure 12 In embodiments, the bitstream description information may include at least one of the following: a first indicator, a second indicator, a third indicator, or a fourth indicator. The first indicator indicates whether the bitstream includes encoded data of a first rendering intermediate result; the second indicator may indicate the type of the first rendering intermediate result; the third indicator may indicate whether the bitstream includes a third rendering parameter; and the fourth indicator may indicate the type of the third rendering parameter.

[0432] For example, in Figure 12 In this embodiment, the first indicator can be a second preset information, indicating that the bitstream does not contain encoded data of the first rendering intermediate result.

[0433] For example, the first intermediate rendering result corresponds to one or more types. Specifically, when the first intermediate rendering result corresponds to one type, the type of the first intermediate rendering result can be written into the bitstream as the value of the second indicator. When the first intermediate rendering result corresponds to multiple types, the type of the first intermediate rendering result used in the encoding process described above can be written into the bitstream as the value of the second indicator.

[0434] For example, when the third indicator is the first preset information (such as 1), it means that the bitstream contains the third rendering parameter; when the third indicator is the second preset information (such as 0), it means that the bitstream does not contain the third rendering parameter.

[0435] For example, the parameters included in the second rendering parameters can correspond to multiple types. For instance, the motion information of a rigid dynamic object corresponds to type C1, the motion information of a non-rigid dynamic object corresponds to type C2, and so on. The type of the third rendering parameter written to the bitstream can be written into the bitstream as the value of the fourth indicator.

[0436] The following example uses the bitstream description information, which includes the first indicator, second indicator, third indicator, and fourth indicator, to illustrate... Figure 12 The decoding process corresponding to the encoding process will be explained.

[0437] Figure 13 This is a schematic diagram illustrating the decoding process as an example. Figure 13 What is shown is the same as Figure 12 The corresponding decoding process. In Figure 13In an embodiment, the terminal device performs rendering processing based on the first rendering parameter generated locally and the third rendering parameter received from the server, and generates a first rendering intermediate result; wherein the third rendering parameter comprises part of the second rendering parameter. Figure 13 In an embodiment, taking decoding a frame of image as an example for illustration.

[0438] In S1301, the terminal device receives a code stream.

[0439] In S1302, the terminal device parses the code stream to obtain a parsing result, the parsing result comprising a residual block corresponding to the current frame and a third rendering parameter corresponding to the current frame.

[0440] In S1303, the terminal device generates a first rendering parameter and a fourth rendering parameter.

[0441] For example, the first rendering parameter comprises a camera parameter. The camera parameter can comprise but is not limited to a view projection matrix, a camera dynamic range, a focal length, an aperture size, a depth of field, and the like.

[0442] For example, the terminal device can parse a third indication identifier from the code stream, and when it is determined based on the third indication identifier that the third rendering parameter is included in the code stream, the type of the third rendering parameter can be determined according to a fourth indication identifier. Then, the fourth rendering parameter is generated according to the type of the third rendering parameter.

[0443] For example, when the server writes part of the second rendering parameter into the code stream, i.e., the third rendering parameter comprises part of the second rendering parameter, and the fourth indication identifier comprises the type of the part of the second rendering parameter; at this time, the terminal device can generate the fourth rendering parameter based on the type of the third rendering parameter, wherein the fourth rendering parameter is the parameter in the second rendering parameter other than the third rendering parameter.

[0444] For example, assuming that the type of the third rendering parameter is type C1, i.e., the third rendering parameter is the motion information of a rigid dynamic object, the fourth rendering parameter generated by the terminal device can comprise, for example, scene geometry information (such as the vertex coordinates of a triangular patch), material data (such as material maps, material generation functions), and texture data (such as texture maps), and the motion information of a non-rigid dynamic object.

[0445] For example, assuming that the type of the third rendering parameter is type C2, i.e., the third rendering parameter is the motion information of a non-rigid dynamic object, the fourth rendering parameter generated by the terminal device can comprise, for example, scene geometry information (such as the vertex coordinates of a triangular patch), material data (such as material maps, material generation functions), and texture data (such as texture maps), and the motion information of a rigid dynamic object.

[0446] For example, assuming that the type of the third rendering parameter includes type C1 and type C2, i.e., the third rendering parameter is the motion information of the rigid dynamic object and the motion information of the non-rigid dynamic object, the fourth rendering parameter generated by the terminal device can include, for example, scene geometry information (such as the vertex coordinates of a triangular patch), material data (such as material mapping, material generation functions), and texture data (such as texture mapping).

[0447] In S1304, the terminal device performs rendering processing on the three-dimensional scene based on the first rendering parameter, the third rendering parameter, and the fourth rendering parameter, and generates a first rendering intermediate result in the rendering processing.

[0448] For example, when the third rendering parameter includes part of the parameters in the second rendering parameter, the terminal device performs rendering processing on the three-dimensional scene based on the first rendering parameter, the third rendering parameter, and the fourth rendering parameter.

[0449] It should be understood that when the server writes all the second rendering parameters into the code stream, i.e., the third rendering parameter includes all the parameters in the second rendering parameter, the fourth indication identifies the types of all the second rendering parameters; the terminal device can decode all the second rendering parameters. At this time, the terminal device does not need to generate the fourth rendering parameter, and then performs rendering processing on the three-dimensional scene based on the first rendering parameter and the third rendering parameter.

[0450] For example, the first rendering intermediate result can be CGMV, depth information, or a Render ID.

[0451] For example, the terminal device can determine, according to the first indication identified by parsing the code stream, that the code stream does not include the encoded data of the first rendering intermediate result, and in the rendering processing, can generate the first rendering intermediate result of the corresponding type according to the second indication identified by parsing.

[0452] In S1305, the terminal device reconstructs based on the first rendering intermediate result and the residual block to obtain a reconstructed image of the current frame; wherein the first rendering intermediate result acts on at least one processing in the reconstruction process: prediction or filtering.

[0453] For example, S1305 can refer to the description above, which will not be repeated here.

[0454] It should be noted that when the code stream description parameter of the code stream does not include the first indication, Figure 13 In the embodiment of the terminal device can determine whether the code stream includes the encoded data of the first rendering intermediate result according to the pre-agreed information. When the code stream description parameter of the code stream does not include the second indication, Figure 13In the embodiment of the present application, the terminal device can determine the type of the first rendering intermediate result according to the pre-agreed information. When the code stream description parameter of the code stream does not include the third indication identifier, Figure 13 In the embodiment of the present application, the terminal device can determine whether the third rendering parameter is included in the code stream according to the pre-agreed information. When the code stream description parameter of the code stream does not include the fourth indication identifier, Figure 13 In the embodiment of the present application, the terminal device can determine the type of the third rendering parameter in the parsing result according to the pre-agreed information.

[0455] For example, the second rendering parameter has a small data amount (a few / ten kB), so even if the server sends all or part of the second rendering parameter to the terminal device in addition to the second rendering intermediate result, the code rate overhead of the data stream transmitted by the server to the terminal device is also smaller than the code rate overhead of the data stream transmitted by the server to the terminal device in the prior art, and the computing power of the terminal device can also be saved.

[0456] Figure 14 An example encoding process is shown in the schematic diagram. In the embodiment of the present application, Figure 14 In the embodiment of the present application, the server sends the second rendering intermediate result and the third rendering parameter to the terminal device. Figure 14 In the embodiment of the present application, an example of encoding a frame of rendered image is described.

[0457] S1401, the server receives the first rendering parameter sent by the terminal device.

[0458] For example, the first rendering parameter can include camera parameters and motion information of a non-rigid dynamic object (such as the bone nodes of a moving character). The camera parameters can include but are not limited to view projection matrix, camera dynamic range, focal length, aperture size, depth of field, and the like.

[0459] S1402, the server generates the second rendering parameter.

[0460] For example, the server can load the scene geometry information (such as the vertex coordinates of the triangular patches), material data (such as material maps, material generation functions), and texture data (such as texture maps) pre-stored in the memory (such as hard disk, memory, etc.), motion information of dynamic objects, and the like, and then the second rendering parameter can be obtained.

[0461] For example, the motion information of the dynamic object can include motion information (such as motion trajectory) of a rigid dynamic object and / or motion information (such as motion trajectory) of a non-rigid dynamic object (which refers to other non-rigid dynamic objects in addition to the moving character).

[0462] S1403, the server performs rendering processing on the three-dimensional scene based on the first rendering parameter and the second rendering parameter to obtain a rendered image.

[0463] S1404, The server selects the first intermediate rendering result based on the intermediate rendering results generated during the rendering process.

[0464] For example, in Figure 14 In one embodiment, the first intermediate rendering result may be CGMV, depth information, or RenderID.

[0465] S1405, the server encodes the rendered image based on the first intermediate rendering result to obtain the encoded data of the residual block and writes the encoded data of the residual block into the bitstream; the first intermediate rendering result is applied to at least one of the processing steps in the encoding process: block segmentation, prediction, or filtering.

[0466] S1406, the server encodes the second rendering intermediate result and writes the encoded data of the second rendering intermediate result into the bitstream; wherein, the second rendering intermediate result is a part of the first rendering intermediate result.

[0467] S1407, the server writes the third rendering parameter into the bitstream.

[0468] For example, S1405 to S1407 can be described with reference to the above embodiments, and will not be repeated here.

[0469] For example, in Figure 14 In the embodiments, the bitstream description information may include at least one of the following: a first indicator, a second indicator, a third indicator, and a fourth indicator. The first indicator indicates whether the bitstream contains encoded data of a second rendering intermediate result; the second indicator can be used to indicate the type and subtype of the second rendering intermediate result; the third indicator indicates whether the bitstream includes a third rendering parameter; and the fourth indicator indicates the type of the third rendering parameter.

[0470] The following example uses the bitstream description information, which includes the first indicator, second indicator, third indicator, and fourth indicator, to illustrate... Figure 14 The decoding process corresponding to the encoding process will be explained.

[0471] Figure 15 This is a schematic diagram illustrating the decoding process as an example. Figure 15 What is shown is with Figure 14 The corresponding decoding process. In Figure 15 In one embodiment, the terminal device performs rendering processing based on locally generated first rendering parameters and third rendering parameters received from the server, and generates a third intermediate rendering result; wherein, the third rendering parameters include a portion of the second rendering parameters. Figure 15 In this embodiment, decoding a single frame of image is used as an example for illustration.

[0472] S1501, the terminal device receives the bit stream.

[0473] S1502, the terminal device parses the bitstream to obtain the parsing result, which includes the residual block corresponding to the current frame, the second intermediate rendering result corresponding to the current frame, and the third rendering parameter corresponding to the current frame.

[0474] S1503, the terminal device generates the first rendering parameters and the fourth rendering parameters.

[0475] For example, the first rendering parameters include camera parameters and motion information of non-rigid dynamic objects (such as the skeletal nodes of a moving character). Among them, camera parameters may include, but are not limited to, parameters such as view projection matrix, camera dynamic range, focal length, aperture size, and depth of field.

[0476] S1504, the terminal device renders the 3D scene based on the first rendering parameters, the third rendering parameters and the fourth rendering parameters, and generates the third rendering intermediate result based on the second rendering intermediate result during the rendering process.

[0477] S1505, the terminal device performs reconstruction based on the first intermediate rendering result and the residual block to obtain the reconstructed image of the current frame; wherein, the first intermediate rendering result is applied to at least one of the processing steps in the reconstruction process: prediction or filtering.

[0478] For example, S1504 to S1505 can be referred to the description above, and will not be repeated here.

[0479] It should be noted that when the bitstream description parameters do not include the first indicator, Figure 15 In this embodiment, the terminal device can determine whether the bitstream includes encoded data of the second rendering intermediate result based on pre-agreed information. When the bitstream description parameter does not include the second indicator, Figure 15 In this embodiment, the terminal device can determine the type and subtype of the second rendering intermediate result based on pre-agreed information. When the bitstream description parameters do not include a third indicator, Figure 15 In this embodiment, the terminal device can determine whether the bitstream includes a third rendering parameter based on pre-agreed information. When the bitstream description parameter does not include a fourth indicator, Figure 15 In this embodiment, the terminal device can determine the type of the third rendering parameter in the parsing result based on pre-agreed information.

[0480] It should be understood that, Figure 2 , Figure 4 , Figure 6a , Figure 8a , Figure 10a , Figure 12 , Figure 14In an embodiment, the server can also not obtain the first rendering parameter from the terminal device, but generate the first rendering parameter by itself. The first rendering parameter generated by the server includes the same type of parameters as the first rendering parameter generated by the terminal device, and the accuracy of the parameters included in the first rendering parameter generated by the server is greater than or equal to the accuracy of the parameters included in the first rendering parameter generated by the terminal device. For example, the server and the terminal device can generate the first rendering parameter according to a pre-agreement.

[0481] The following describes the process of the first rendering intermediate data acting on at least one of the following processes in the encoding process of the rendered image: blocking, prediction or filtering, and the corresponding decoding process.

[0482] Figure 16a An example of an encoding framework is shown in the schematic diagram. Referring to Figure 16a , for example, the encoder can include a blocking module, an intra prediction module, an inter prediction module, a transform module, a quantization module, an entropy encoding module, a dequantization module, an inverse transform module, a loop filtering module, and a memory. Among them, Figure 16a The blocking module of the existing encoder is optimized to obtain the blocking module. It should be understood that the encoder of the present application can also include the blocking module of the existing encoder, or the blocking module in the encoder of the present application can include all or part of the functions of the blocking module in the existing encoder.

[0483] The following is based on Figure 16a , the process of applying depth information to the blocking of the blocking module to encode the rendered image is described.

[0484] Figure 16b An example of an encoding process is shown in the schematic diagram. Figure 16b The encoding process shown in Figure 16a is based on, Figure 16b The steps in Figure 16b are performed by the encoder; in an embodiment of , the first rendering intermediate result is depth information.

[0485] S1601, based on the depth information, the rendered image is blocked to obtain a plurality of prediction units.

[0486] For example, the to-be-encoded video and the depth information can be input to the blocking module; the blocking module blocks each frame of the rendered image in the to-be-encoded video based on the depth information to obtain a plurality of prediction units. The following describes an example of the blocking module blocking a frame of the rendered image.

[0487] For example, the segmentation module can first divide the rendered image into multiple Coding Tree Units (CTUs) according to the encoder's preset parameters. (See reference...) Figure 16c , Figure 16c (1) is the rendered image. Figure 16c (2) This represents a CTU. Next, for one of the multiple CTUs, it can be divided into multiple coding units (CUs) according to a preset partitioning rule. For example... Figure 16c As shown in (3). Figure 16c (2) One CTU is divided into multiple CUs, and one box corresponds to one CU.

[0488] It should be understood that, in one possible scenario, each CTU is divided into multiple CUs. In another possible scenario, some CTUs are not divided into CUs, while each CTU in another set of CTUs is divided into multiple CUs; ​​the specific division is determined according to the preset division rules and the image content of the CTUs. For details on the division of CTUs into CUs in existing technologies, please refer to the descriptions in the prior art, which will not be repeated here.

[0489] The following explanation uses the example of dividing one of multiple Control Units (CUs) into multiple Prediction Units (PUs). For instance, the CU can be divided based on depth information to obtain multiple PUs. For example, Computer Graphic Edge (CGE) information can be determined based on the depth information; then, the CU can be divided into multiple PUs based on the CGE information. The CGE information can include object edge information of objects in the rendered image. Based on the CGE information, the CU can be reasonably divided into multiple PUs, resulting in better prediction performance (e.g., more accurate prediction blocks). Better prediction performance leads to smaller residual blocks, reducing the bitrate and improving image reconstruction quality. Furthermore, compared to existing PU division methods, dividing PUs based on CGE information requires less computation.

[0490] Figure 16d This is a schematic diagram illustrating the CGE information generation process as an example. Figure 16d (1) is the rendered image. Figure 16d (2) Depth information (also known as a depth map). Specifically, the depth information can be filtered (e.g., Sobel filtering) to obtain an edge depth map, such as... Figure 16d (3) is shown. Next, clustering operations can be performed on the edge grayscale image (e.g., using Otsu's method) to obtain the edge binary image, as shown. Figure 16d(4) is shown. After that, the CGE information can be generated based on the edge binary map. In one possible implementation, the edge binary map can be used as the CGE information, that is, the CGE information is the edge binary map. Figure 16d (4) is shown. After that, the CGE information can be generated based on the edge binary map. In one possible implementation, the edge binary map can be used as the CGE information, that is, the CGE information is the edge binary map. Figure 16d (4) is shown. After that, the CGE information can be generated based on the edge binary map. In one possible implementation, the edge binary map can be used as the CGE information, that is, the CGE information is the edge binary map. Figure 16d (4) is shown. After that, the CGE information can be generated based on the edge binary map. In one possible implementation, the edge binary map can be used as the CGE information, that is, the CGE information is the edge binary map.

[0491] It should be understood that other filtering algorithms can be used to filter the depth information, which are not limited in the present application. In addition, other clustering algorithms can be used to perform clustering operation on the edge grayscale map, which are not limited in the present application.

[0492] In one possible implementation, the object edge lines in the CU can be determined based on the CGE information; then, a plurality of candidate partition paths can be searched from the object edge lines; after that, the CU can be divided into a plurality of PUs according to the target partition path selected from the plurality of candidate partition paths.

[0493] Specifically, assuming that the CGE information is the edge binary map, the target region matching the CU can be determined from the edge binary map based on the region edge coordinates of the CU in the rendered image. The region edge coordinates of the target region in the edge binary map are the same as the region edge coordinates of the CU in the rendered image. Then, a depth-first search method can be used to find all paths (it should be understood that the path is not a pixel, but a boundary line between two pixels) that can divide the target region into two sub-regions from the object edge lines (i.e., the region with pixel value of 1) of the target region, that is, the candidate partition paths. After that, the shortest candidate partition path can be selected from the candidate partition paths as the target partition path. Subsequently, the CU can be divided into a plurality of PUs according to the target partition path.

[0494] Figure 16e The PU partition process is shown for example. Wherein, Figure 16e (1) is a CTU, Figure 16e (2) is an edge binary map, and the CGE information is the edge binary map.

[0495] Referring to Figure 16e For a CU (such as CU1) in the CTU, the edge binary map (such as edge binary map 1) is generated based on the CGE information (such as CGE information 1) of the CU. Figure 16e(2) is shown. The target region matched with the object edge line shown in (1) is shown as R2. The region edge coordinates of R2 are the same as the region edge coordinates of CU1. Assuming that the width of the object edge line in the R2 region (i.e., the white region in R2) is 4 pixels, 5 candidate splitting paths can be found. Then, one of the 5 candidate splitting paths can be selected as the target splitting path. Then, CU1 can be divided into two PUs, as shown in (3). Figure 16e (3) shows that CU1 is divided into PU1 and PU2. By analogy, Figure 16e (1), other CUs can also be divided into multiple PUs in this manner, as shown in (3). Figure 16e (3).

[0496] It should be understood that when the object edge line in the target region includes multiple lines, one CU can be divided into 2 or more PUs.

[0497] It should be understood that in one possible case, each CU is divided into multiple PUs. In one possible case, part of the CUs are not divided into PUs, and each of the other CUs is divided into multiple PUs.

[0498] It should be understood that one prediction unit can be used as a to-be-encoded block, a CTU that is not divided into CUs can be used as a to-be-encoded block, and a CU that is not divided into PUs can be used as a to-be-encoded block.

[0499] S1602, predicting the multiple prediction units based on the reconstructed blocks to obtain multiple prediction blocks, the multiple prediction blocks corresponding to the multiple prediction units one by one.

[0500] For example, when the rendered image is a key frame, for each of the multiple PUs, the intra prediction module can perform intra prediction based on the reconstructed block corresponding to the rendered image to determine the prediction block corresponding to the PU; in this way, multiple prediction blocks can be determined for the multiple PUs, and the multiple prediction blocks correspond to the multiple prediction units one by one.

[0501] For example, when the rendered image is a non-key frame, a reference frame of the rendered image can be determined; for each of the multiple PUs, the inter prediction module can perform inter prediction based on the reference frame (the reference frame includes multiple reconstructed blocks) corresponding to the rendered image to determine the prediction block corresponding to the PU; in this way, multiple prediction blocks can be determined for the multiple PUs, and the multiple prediction blocks correspond to the multiple prediction units one by one.

[0502] S1603, encoding multiple residual blocks between the multiple prediction blocks and the multiple prediction units to obtain encoding data of the multiple residual blocks, the multiple residual blocks corresponding to the multiple prediction units one by one.

[0503] For example, for each of the multiple prediction blocks (PUs), the PU can be residuald with the corresponding prediction block to obtain a residual block for that PU; thus, multiple residual blocks can be obtained. Next, these multiple residual blocks can be encoded to obtain encoded data; then, the encoded data of these multiple residual blocks can be written into the bitstream.

[0504] Refer again Figure 16a For example, for each residual block, the residual block is input to the transform module, which transforms the residual block to obtain the transformed result and outputs the transformed result to the quantization module. Subsequently, the quantization module can quantize the transformed result to obtain the quantized result and output the quantized result to the entropy coding module; then, the entropy coding module can entropy code the quantized result to obtain the encoded data of the residual block and write the encoded data of the residual block into the bit stream.

[0505] For example, the quantization module can also output the quantization result to the inverse quantization module, which performs inverse quantization to obtain the inverse quantization result and outputs it to the inverse transform module. Next, the inverse transform module performs an inverse transform on the inverse quantization result to obtain a decoded residual block. Then, the decoded residual block can be superimposed with the prediction block output by the intra-frame prediction module to obtain a reconstructed block. Subsequently, the reconstructed block is input to the loop filtering module, which performs loop filtering on the reconstructed block to obtain a filtered reconstructed block and outputs it to the memory. (The loop filtering module, after obtaining the filtered reconstructed block, can stitch the filtered reconstructed block to the corresponding position in the image to be reconstructed; after obtaining a frame of reconstructed image, it can output the reconstructed image to the memory for storage.)

[0506] It should be understood that predictions can also be made for CTU and CU according to S1602 to S16031, and the residual blocks between CTU and the corresponding prediction blocks and between CU and the corresponding prediction blocks can be encoded, which will not be elaborated here.

[0507] Figure 17a This is a schematic diagram illustrating an exemplary decoding framework. (Refer to...) Figure 17a For example, the decoder of this application may include: an entropy decoding module, an inverse quantization module, an inverse transform module, an intra-frame prediction module, an inter-frame prediction module, a loop filtering module, and a memory. Among these, Figure 17aThe inter prediction module of the present application is optimized based on the inter prediction module of the existing encoder, and the intra prediction module of the present application is optimized based on the intra prediction module of the existing encoder. It should be understood that the inter prediction module of the present application can also include all or part of the functions of the inter prediction module of the existing encoder, or the present application can also include the inter prediction module of the existing encoder; and the intra prediction module of the present application can also include all or part of the functions of the intra prediction module of the existing encoder, or the present application can also include the intra prediction module of the existing encoder.

[0508] The following is based on Figure 17a The process of applying depth information to the prediction of the inter prediction module / intra prediction module to decode the rendered image is described.

[0509] Figure 17b The decoding process is schematically shown as an example. Figure 17b The decoding process is based on Figure 17a The steps in Figure 17b are performed by the decoder; in Figure 17b an embodiment, the first rendered intermediate result is depth information.

[0510] S1701, based on the depth information, determine the blocking information of the current frame.

[0511] For example, after receiving the code stream, the terminal device can parse the code stream to obtain a parsing result. The parsing process can be as follows: the coding data of the residual block is extracted from the code stream, and then the coding data of the residual block is input to the entropy decoding module, and the entropy decoding module is used to entropy decode the coding data of the residual block to obtain entropy decoding data. Then, the entropy decoding data can be input to the inverse quantization module, and the inverse quantization module is used to perform inverse quantization to obtain inverse quantization data and input the inverse quantization data to the inverse transform module. Subsequently, the inverse transform module performs inverse transform on the inverse quantization data to obtain the residual block of the current frame (i.e. Figure 17a the decoded residual block shown in FIG. 17).

[0512] It should be noted that the code stream description information used to describe the code stream can also be extracted from the code stream, at which time, the code stream description information does not need to be entropy decoded, inverse quantized, and inverse transformed, etc.

[0513] For example, when it is determined to be intra prediction according to the code stream description information parsed from the code stream, the intra prediction module can be used to perform intra prediction to obtain a prediction block, i.e., S1701-S1702 are performed.

[0514] For example, when it is determined to be an inter-frame prediction based on the bitstream description information parsed from the bitstream, the intra-frame prediction module can perform inter-frame prediction to obtain the prediction block, i.e., execute S1701 to S1702.

[0515] For example, the partitioning methods of CTU and CU can be determined first; then, CGE information can be determined based on depth information; then, the method of dividing the CU into multiple PUs (i.e., the PU partitioning method) can be determined based on the CGE information; finally, the partitioning methods of CTU, CU, and PU are determined as the block information of the current frame. The partitioning methods of CTU, CU, and PU can be referred to in the description of S1601 above, and will not be repeated here.

[0516] S1702. Based on the block information and reconstruction blocks, predict the multiple prediction units included in the current frame to obtain multiple prediction blocks. The multiple prediction blocks correspond one-to-one with the multiple prediction units, and the multiple residual blocks correspond one-to-one with the multiple prediction units.

[0517] For example, when it is determined to be intra-frame prediction based on the bitstream description information parsed from the bitstream, the intra-frame prediction module can predict multiple prediction units included in the current frame based on block information and the reconstruction blocks corresponding to the rendered image to obtain multiple prediction blocks.

[0518] For example, when it is determined to be an inter-frame prediction based on the bitstream description information parsed from the bitstream, the inter-frame prediction module can predict multiple prediction units included in the current frame based on the block information and the reference frame corresponding to the rendered image (the reference frame includes multiple reconstructed blocks) to obtain multiple prediction blocks.

[0519] In this system, multiple prediction blocks correspond one-to-one with multiple prediction units, and multiple residual blocks correspond one-to-one with multiple prediction units.

[0520] S1703. Based on multiple residual blocks and multiple prediction blocks, determine the reconstructed image of the current frame.

[0521] For example, for each residual block, the residual block ( Figure 17a The decoded residual block shown in the diagram is superimposed with the corresponding predicted block to obtain a reconstructed block. Then, a loop filter can be applied to the reconstructed block to obtain a filtered reconstructed block, which is then output to a memory. (The loop filter module, after obtaining the filtered reconstructed block, can stitch it to the corresponding position in the image to be reconstructed; after obtaining a frame of reconstructed image, it can output the reconstructed image to a memory for storage.) In this way, multiple frames of reconstructed images can be combined to form a reconstructed video.

[0522] It should be noted that when the first rendering intermediate result is depth information, the server can not transmit the second rendering intermediate result, but transmit CGE information. Wherein, the CGE information can be encoded, and then the encoding data of the CGE information is written into the code stream. In this case, the terminal device can not need to generate depth information, and at this time, S1701 can be replaced by determining the blocking information of the current frame based on the CGE information.

[0523] In Figure 16b or Figure 17b Embodiments, the CGE information determined based on the depth information includes object edge information of the object in the rendered image; based on the CGE information, the CU can be reasonably divided into multiple PUs, and then the subsequent prediction effect (for example, the prediction block obtained by subsequent prediction is more accurate) is better; when the prediction effect is better, the determined residual block is smaller, which can reduce the code rate; and can also improve the reconstruction quality of the image. In addition, compared with the existing PU partitioning method, the calculation amount of the PU partitioning based on the CGE information is smaller.

[0524] Figure 18a The encoding framework is schematically shown as an example. Referring to Figure 18a , for example, the encoder can include a blocking module, an intra prediction module, a first inter prediction module, a second inter prediction module, a transform module, a quantization module, an entropy encoding module, a dequantization module, an inverse transform module, a loop filtering module and a memory. Wherein, Figure 18a The second inter prediction module of the second inter prediction module is the inter prediction module in the existing encoder, or in other words, the second inter prediction module includes all or part of the functions of the inter prediction module in the existing encoder, and the first inter prediction module is the newly added inter prediction module relative to the existing encoder.

[0525] The following is based on Figure 18a , the process of applying CGMV to the prediction of the first inter prediction module to encode the rendered image is described.

[0526] Figure 18b The encoding process is schematically shown as an example. Figure 18b is shown based on Figure 18a , the encoding process, Figure 18b The steps in the process are performed by the encoder; in Figure 18b The encoding process of Figure 18b is an inter coding process.

[0527] S1801, based on the reconstructed block and the CGMV, the prediction of the rendered image in the to-be-encoded block is performed to obtain the prediction block corresponding to the to-be-encoded block.

[0528] For example, the video to be encoded can be input into a block division module; each frame of the rendered image in the video to be encoded is divided into a plurality of blocks to be encoded by the block division module. Then, inter-frame prediction is performed based on the CGMV by a first inter-frame prediction module.

[0529] For example, a reference frame of the rendered image is determined from the reconstructed image (the reference frame includes a plurality of reconstructed blocks); then, one or more reference blocks (the reference block can be a complete reconstructed block, can be a part of the reconstructed block, and can also be composed of a plurality of reconstructed blocks) are selected from the reference frame of the rendered image according to the region edge coordinates of the block to be encoded in the rendered image; wherein the size of the reference block is the same as the size of the block to be encoded. For example, a target image block in the reference frame of the rendered image is determined according to the region edge coordinates of the block to be encoded in the rendered image, and the region edge coordinates of the target image block in the reference frame are the same as the region edge coordinates of the block to be encoded in the rendered image. Then, one or more other image blocks around the target image block are selected as the reference block; for example, the other image blocks around the target image block are selected as the reference block.

[0530] Then, a prediction block matched with the block to be encoded can be determined based on the reference block of the block to be encoded and the CGMV (the CGMV between the rendered image and the reference frame, including the CGMV of each pixel point in the reference frame). When the reference block of the block to be encoded is a plurality of reference blocks, for one reference block, the CGMV corresponding to the reference block (including the CGMV of each pixel point in the reference block) can be selected from the CGMV between the rendered image and the reference frame according to the pixel point coordinates of the reference block. Then, a prediction block is determined based on the reference block and the CGMV corresponding to the reference block.

[0531] For example, the pixel points of the reference block can be moved based on the CGMV corresponding to the reference block to obtain a prediction pixel set. Then, the prediction block is generated based on the prediction pixel set and the pixel values of the pixel points in the reference block. Specifically, for a first pixel point in the prediction pixel set, if the first pixel point is out of the boundary of the reference block, in one possible manner, the brightness of the first pixel point can be set to a first preset value, and the chroma value can be set to a second preset value. The first preset value can be set as 16 according to requirements, and the second preset value can be set as 128 according to requirements; the present application does not make any limitation thereon. In one possible manner, the pixel value of the first pixel point can be set to the average pixel value of the reference block.

[0532] For example, if the first pixel point is not out of the boundary of the reference block, the pixel value of a second pixel point in the reference block corresponding to the position of the first pixel point is obtained, and the pixel value of the second pixel point is taken as the pixel value of the first pixel point.

[0533] In a possible implementation, when the plurality of prediction blocks are determined, rate-distortion costs corresponding to the plurality of prediction blocks can be calculated, and the prediction block with the minimum rate-distortion cost can be selected to perform S1802.

[0534] In a possible implementation, when the plurality of prediction blocks are determined, the plurality of prediction blocks can be fused to obtain a first fused prediction block; then, rate-distortion costs corresponding to the plurality of prediction blocks and the rate-distortion cost of the first fused prediction block can be calculated, and the prediction block with the minimum rate-distortion cost can be selected to perform S1802.

[0535] For example, the plurality of prediction blocks can be fused according to the following formula:

[0536] U = (ω1*V1 + ω2*V2... + ωn*Vn) / (ω1 + ω2... + ωn)

[0537] wherein V1, V2...Vn are the prediction blocks, U is the first fused prediction block, and n is the number of prediction blocks. ω1, ω2...ωn are weights corresponding to the prediction blocks, which can be set according to requirements, and the application does not limit this.

[0538] In a possible implementation, on the basis of S1801, the second inter-frame prediction module can also perform traditional inter-frame prediction (that is, inter-frame prediction in the prior art) based on the reference frame of the rendered image, to obtain one or more prediction blocks. For ease of description, the prediction block determined based on CGMV can be referred to as a first prediction block, and the prediction block determined based on the traditional inter-frame prediction manner can be referred to as a second prediction block.

[0539] In this case, in a possible implementation, the prediction block with the minimum rate-distortion cost can be selected from the first prediction block and the second prediction block to perform S1802. In a possible implementation, when there are a plurality of second prediction blocks, the second prediction blocks can be fused to obtain a second fused prediction block; then, the prediction block with the minimum rate-distortion cost can be selected from the second prediction blocks, the second fused prediction block, the first prediction block, and the first fused prediction block to perform S1802. In a possible implementation, the second prediction blocks and the first prediction block can also be fused to obtain a third fused prediction block; then, the prediction block with the minimum rate-distortion cost can be selected from the first prediction block, the second prediction block, and the third fused prediction block to perform S1802; as shown in Figure 18c Figure 18c In the above implementations, the first inter-frame prediction is performed by the first inter-frame prediction module, and the second inter-frame prediction is performed by the second inter-frame prediction module.

[0540] ​In one possible approach, S1802 can also be performed by selecting the prediction block with the lowest rate-distortion cost from among the first prediction block, the first fused prediction block, the second prediction block, the second fused prediction block, and the third fused prediction block; it should be understood that this application does not limit the method of selecting the prediction block ultimately used to perform S1802.

[0541] S1802, the residual block between the block to be encoded and the corresponding prediction block in the rendered image is encoded to obtain the encoded data of the residual block.

[0542] For example, for each of the multiple blocks to be encoded, the block to be encoded can be residuald with the corresponding residual block to obtain the residual block corresponding to the block to be encoded; thus, multiple residual blocks can be obtained. Next, these multiple residual blocks can be encoded to obtain encoded data; then, the encoded data of these multiple residual blocks is written into the bitstream. The specific process of encoding the residual blocks can be referred to the description of S1603 above, and will not be repeated here.

[0543] For example, the code to be encoded could be a PU.

[0544] For example, when the prediction block in S1802 is a prediction block determined based on CGMV, the block identifier of the reference block used to determine the corresponding prediction block can be written into the bitstream. When the prediction block in S1802 is a prediction block determined based on a traditional inter-frame prediction method, the frame identifier of the reference frame and the motion vector between the prediction block and the block to be encoded can be written into the bitstream. Both the block identifier and the frame identifier can be referred to as bitstream description information.

[0545] Figure 19a This is a schematic diagram illustrating an exemplary decoding framework. (Refer to...) Figure 19a For example, the encoder of this application may include: an entropy decoding module, an inverse quantization module, an inverse transform module, an intra-frame prediction module, a first inter-frame prediction module, a second inter-frame prediction module, a loop filtering module, and a memory. Among these, Figure 19a The second inter-frame prediction module is the inter-frame prediction module in the existing encoder, or in other words, the second inter-frame prediction module includes all or part of the functions of the inter-frame prediction module in the existing encoder, while the first inter-frame prediction module is an inter-frame prediction module added relative to the existing encoder.

[0546] The following is based on Figure 19a This paper explains the process of applying CGMV to the inter-frame prediction module to decode the rendered image.

[0547] Figure 19b This is a schematic diagram illustrating the decoding process as an example. Figure 19b Therefore Figure 19a The decoding process is based on this. Figure 19bthe steps in the method are performed by a decoder; in Figure 19b In an embodiment of the method, the first rendering intermediate result is CGMV.

[0548] S1901, predicting the to-be-decoded block in the current frame based on the CGMV and the reconstructed block to obtain a prediction block corresponding to the to-be-decoded block.

[0549] For example, when it is determined according to the stream description information parsed from the stream that the prediction block in the encoding process is determined based on CGMV, S1901 can be performed. For example, for the to-be-decoded block in the current frame, the block identifier of the stream description information can be used to select a reconstructed block matching the to-be-decoded block from the reference frame corresponding to the current frame (the reference frame includes a plurality of reconstructed blocks); then, based on the CGMV and the reconstructed block matching the to-be-decoded block, the prediction block corresponding to the to-be-decoded block is generated, and details can be referred to the description of S1801 above, which will not be described here.

[0550] For example, when it is determined according to the stream description information parsed from the stream that the prediction block in the encoding process is determined based on the traditional inter-frame prediction method, the prediction block determined by the traditional inter-frame prediction method can be used at the decoding end; details will not be described here.

[0551] S1902, determining the reconstructed image of the current frame based on the residual block and the prediction block.

[0552] For example, for each residual block, the residual block (decoded residual block shown in FIG. 1) can be superimposed with the prediction block corresponding to the residual block to obtain a reconstructed block. Then, the reconstructed block can be loop filtered to obtain a filtered reconstructed block and output the filtered reconstructed block to the memory. (Wherein, after obtaining the filtered reconstructed block, the loop filtering module can splice the filtered reconstructed block to the corresponding position of the to-be-reconstructed image; after obtaining a frame of reconstructed image, the reconstructed image can be output to the memory for storage.) In this way, a plurality of reconstructed images can constitute a reconstructed video. Figure 19a In an embodiment of the method,

[0553] and Figure 18b In an embodiment of the method, the CGMV is generated by a graphics method, which avoids the problem of inaccurate motion estimation of the existing encoder; and the CGMV is a pixel-level MV, while the MV generated by the existing encoder is an image block-level MV. The pixel-level MV can more accurately depict the edges of the object and reduce the prediction error; therefore, determining the prediction block based on the CGMV can reduce the error of the prediction block to a certain extent, improve the accuracy of the prediction block, and then improve the inter-frame encoding compression efficiency. Figure 19b

[0554] An example of an encoding framework is shown in the figure. Refer to Figure 20a Figure 20a ​The exemplary encoder can include a blocking module, an intra prediction module, an inter prediction module, a transform module, a quantization module, an entropy coding module, a dequantization module, an inverse transform module, a loop filtering module, and a memory. Among them, Figure 20a The loop filtering module in the present application is optimized based on the loop filtering module in the existing encoder. It should be understood that the loop filtering module of the encoder in the present application can also include the existing loop filtering module, or the encoder in the present application also includes all or part of the functions of the loop filtering module in the existing encoder; the present application does not limit this.

[0555] The following is based on Figure 20a The process of applying the Render ID to the filtering of the loop filtering module to encode the rendered image is described below.

[0556] Figure 20b The encoding process is schematically shown as an example. Figure 20b It is shown based on Figure 20a The steps in Figure 20b are performed by the encoder; in the process of Figure 20b , the first rendering intermediate result is the Render ID.

[0557] Referring to Figure 20a and in combination with the above description, the reconstruction process can be as follows: the quantization result is output by the quantization module to the dequantization module, and the dequantization is performed by the dequantization module to obtain the dequantization result and output the dequantization result to the inverse transform module. Next, the inverse transform module performs inverse transform on the dequantization result to obtain the decoded residual block. Then, the decoded residual block can be superimposed with the prediction block output by the intra prediction module to obtain the reconstructed block. Subsequently, the reconstructed block is input to the loop filtering module, and the loop filtering module performs loop filtering on the reconstructed block to obtain the filtered reconstructed block and output the filtered reconstructed block to the memory. (Wherein, after obtaining the filtered reconstructed block, the loop filtering module can splice the filtered reconstructed block to the corresponding position of the image to be reconstructed; after obtaining a frame of reconstructed image, the reconstructed image can be output to the memory for storage). Exemplarily, the process of filtering a reconstructed block by the loop filtering module can refer to the following S2001-S2002:

[0558] S2001, based on the Render ID, determine the filtering parameters.

[0559] Exemplarily, for ease of description, the reconstructed block to be filtered (that is, the reconstructed block in Figure 20a ) can be referred to as the first reconstructed block, and the filtered reconstructed block adjacent to the first reconstructed block can be referred to as the second reconstructed block.

[0560] Exemplarily, the loop filter module can obtain a Render ID map corresponding to the first reconstructed block; wherein the Render ID map corresponding to the first reconstructed block can be a Render ID map corresponding to the to-be-reconstructed image of the first reconstructed block. Then, a first region corresponding to the first reconstructed block in the Render ID map corresponding to the first reconstructed block can be determined; wherein the position and size of the first reconstructed block in the to-be-reconstructed image are the same as the position and size of the first region in the Render ID map. And a second region corresponding to the second reconstructed block in the Render ID map corresponding to the first reconstructed block can be determined; wherein the position and size of the second reconstructed block in the to-be-reconstructed image are the same as the position and size of the second region in the Render ID map. Subsequently, whether the pixels on both sides of the boundary line between the first reconstructed block and the second reconstructed block belong to the same object can be determined according to the color values of the pixels on both sides of the boundary line of the first region (i.e. the edge pixels). Then, the filter parameter can be determined according to the determination result. Wherein the filter parameter can be a boundary strength (BS).

[0561] Exemplarily, if the pixels on both sides of the boundary line between the first reconstructed block and the second reconstructed block belong to the same object, and one of the first reconstructed block and the second reconstructed block is intra predicted, the BS can be set to a first preset parameter value such as 2.

[0562] Exemplarily, if the pixels on both sides of the boundary line between the first reconstructed block and the second reconstructed block belong to the same object, and neither of the first reconstructed block and the second reconstructed block is intra predicted, the BS can be set to a second preset parameter value such as 1 when any one of the following conditions is met. Wherein, assuming that the block composed of the edge pixels on the first reconstructed block among the pixels on both sides of the boundary line between the first reconstructed block and the second reconstructed block is A1, and the block composed of the edge pixels on the second reconstructed block is A2:

[0563] (1) The transform coefficient of A1 is a non-zero transform coefficient or the transform coefficient of A2 is a non-zero transform coefficient, and A1 or A2 is the boundary of a TU (Transform Unit).

[0564] (2) The absolute difference of the motion vector between A1 and A2 is greater than an integer pixel.

[0565] (3) The inter prediction block corresponding to A1 (i.e. the prediction block determined by inter prediction) and the reference frame of the inter prediction block corresponding to A1 are different.

[0566] (4) The number of motion vectors of A1 is different from the number of motion vectors of A2.

[0567] For example, if the pixels on both sides of the boundary line of the first reconstructed block and the second reconstructed block do not belong to the same object, the BS can be set to a third preset parameter value, such as 0.

[0568] Compared with the prior art, the present application introduces the Render ID to determine whether the boundary pixels of the current reconstructed block to be filtered and the adjacent filtered reconstructed block belong to the same object, and then determines the filtering parameter according to the determination result, so that the strength of the loop filtering can be more accurately estimated, and the quality of the filtered reconstructed block can be improved.

[0569] S2002, filtering the reconstructed block based on the filtering parameter.

[0570] For example, when the BS is equal to the first preset parameter value, it indicates that the blocky artifact is obvious, and at this time, the intensity of the boundary of the first reconstructed block (that is, the intensity of the pixels included in A1) can be filtered to make the pixels (that is, the pixels included in A1) on one side of the boundary line of the first reconstructed block more blurred. In addition, the color difference boundary (that is, the color difference of the pixels included in A1) of the first reconstructed block can be filtered according to a preset color difference filtering method.

[0571] For example, when the BS is equal to the second preset parameter value, the color difference boundary of the first reconstructed block can not be filtered, and whether the intensity of the first reconstructed block is filtered can be determined according to the following method: the texture information can be determined according to the change of the pixel value of A1 and the change of the pixel value of A2; then, the QP (quality parameter) value (that is, the quantization parameter when the quantization module is quantized) of A1 and A2 is looked up to obtain a threshold β. If the texture information is less than β, the filtering is performed, otherwise the filtering is not performed. When the texture information is less than β, according to the change of the pixel value of A1 and A2 and the threshold β, it is determined whether the intensity of the first reconstructed block is weakly filtered or strongly filtered; for details, refer to the description in the prior art, which will not be repeated here.

[0572] For example, when the BS is equal to the second preset parameter value, the first reconstructed block can not be filtered.

[0573] S2003, predicting the to-be-encoded block in the rendered image based on the reconstructed block to obtain a prediction block corresponding to the to-be-encoded block, the reconstructed block being a reconstructed block filtered based on the Render ID.

[0574] For example, if the rendered image is inter-predicted, a reference frame (the reference frame including a plurality of reconstructed blocks filtered based on the Render ID) can be selected from the reconstructed image; for the to-be-encoded block in the rendered image, inter-prediction can be performed on the to-be-encoded based on the reference frame to select a prediction block matching the to-be-encoded block from the reference frame.

[0575] For example, if the rendered image is intra predicted, for a to-be-encoded block in the rendered image, a reconstructed block matching the to-be-encoded block can be selected from the reconstructed block corresponding to the rendered image (the reconstructed block is a reconstructed block filtered based on the Render ID) as the prediction block matching the to-be-encoded block.

[0576] S2004, encode the residual block between the to-be-encoded block and the prediction block to obtain the encoded data of the residual block.

[0577] For example, S2004 can refer to the description above, and will not be described here again.

[0578] Figure 21a The decoding framework is schematically shown as an example. Figure 20a The encoding framework is schematically shown as an example. Refer to Figure 21a For example, the encoder can include an intra prediction module, an inter prediction module, an inverse transform module, an inverse quantization module, an entropy decoding module, a loop filtering module, and a memory. Wherein, Figure 21a The loop filtering module of the encoder is optimized based on the loop filtering module in the existing encoder, and it should be understood that the loop filtering module of the encoder of the present application can also include all or part of the functions of the existing loop filtering module, or the encoder of the present application can include the loop filtering module in the existing encoder.

[0579] The following is based on Figure 21a The process of applying the Render ID to the filtering of the loop filtering module to decode the rendered image is described.

[0580] Figure 21b The decoding process is schematically shown as an example. Figure 21b The decoding process is based on Figure 20a The decoding process. Figure 21b The steps in the above are performed by the decoder; in Figure 21b In an embodiment of the above, the first rendering intermediate result is the Render ID.

[0581] S2101, determine the filtering parameter based on the Render ID.

[0582] S2102, filter the reconstructed block based on the filtering parameter.

[0583] For example, S2101-S2102 can refer to the description of S2001-S2002 above, and will not be described here again.

[0584] S2103, predict the to-be-decoded block in the current frame based on the reconstructed block to obtain the prediction block corresponding to the to-be-decoded block.

[0585] For example, if the current frame is inter-predicted, the reference frame of the current frame can be selected from the reconstructed images (including the plurality of reconstructed blocks filtered based on the Render ID), and the to-be-decoded block of the current frame can be inter-predicted based on the reference frame to select a prediction block corresponding to the to-be-decoded block from the reference frame.

[0586] For example, if the current frame is intra-predicted, the to-be-decoded block in the current frame can be selected from the reconstructed block (the reconstructed block filtered based on the Render ID) corresponding to the current frame, as a prediction block matched with the to-be-decoded block.

[0587] S2104, determining the reconstructed image of the current frame based on the residual block and the prediction block.

[0588] For example, S2104 can refer to the description above, which will not be repeated here.

[0589] Compared with the prior art, filtering the reconstructed block based on the first rendering intermediate result can reduce the block effect between the reconstructed blocks, thereby improving the quality of the reconstructed blocks. In this way, encoding the to-be-encoded in the rendered image based on the filtered reconstructed block as a reference can reduce the code rate.

[0590] It should be understood that the first rendering intermediate result can act on any two processing in the encoding process: blocking, prediction or filtering. For example, the first rendering intermediate result acts on the blocking and prediction in the encoding process, and the encoding process for the rendered image can combine the embodiments of Figure 16b and the embodiments of Figure 18b ; and the decoding process for the rendered image can combine the embodiments of Figure 17b and Figure 19b . For example, the first rendering intermediate result acts on the filtering and blocking in the encoding process, and the encoding process for the rendered image can combine the embodiments of Figure 16b and the embodiments of Figure 20b ; and the decoding process for the rendered image can combine the embodiments of Figure 17b and Figure 21b . For example, the first rendering intermediate result acts on the filtering and prediction in the encoding process, and the encoding process for the rendered image can combine the embodiments of Figure 20b and the embodiments of Figure 18b ; and the decoding process for the rendered image can combine the embodiments of Figure 21b and Figure 19b .

[0591] It should be understood that the first rendering intermediate result can act on three processes in the encoding process: blocking, prediction and filtering; at this time, for the encoding process of the rendered image, the embodiments of Figure 16b , the embodiments of Figure 18b and the embodiments of Figure 20b are combined; and for the decoding process of the rendered image, the embodiments of Figure 17b , Figure 19b and Figure 21b are combined.

[0592] In one example, Figure 22 shows a schematic block diagram of an apparatus 2200 of an embodiment of the present application. The apparatus 2200 can include a processor 2201 and a transceiver / transceiver pin 2202, and optionally further include a memory 2203.

[0593] The various components of the apparatus 2200 are coupled together by a bus 2204, which can include a data bus, a power bus, a control bus, and a state signal bus. However, for the sake of clarity, the various buses are shown as a bus 2204.

[0594] Optionally, the memory 2203 can be used to store instructions in the foregoing method embodiments. The processor 2201 can be used to execute the instructions in the memory 2203, and control the receiving pin to receive signals and the sending pin to send signals.

[0595] The apparatus 2200 can be an electronic device or a chip of an electronic device in the above-described method embodiments.

[0596] All relevant content of each step involved in the above method embodiments can be cited to the functional description of the corresponding functional module, which will not be repeated here.

[0597] The embodiment also provides a computer readable storage medium, which stores computer instructions, when the computer instructions run on an electronic device, the electronic device executes the above-mentioned related method steps to realize the encoding and decoding method in the above-mentioned embodiment.

[0598] The embodiment also provides a computer program product, when the computer program product runs on a computer, the computer executes the above-mentioned related steps to realize the encoding and decoding method in the above-mentioned embodiment.

[0599] In addition, the embodiment of the present application further provides a device, which can be a chip, a component or a module, and the device can include a processor and a memory connected to each other; the memory is used to store computer-executable instructions; when the device is running, the processor can execute the computer-executable instructions stored in the memory, so that the chip executes the coding and decoding method in each method embodiment described above.

[0600] Among them, the electronic device, computer readable storage medium, computer program product or chip provided by the embodiment are used to execute the corresponding method provided above, so the beneficial effects that can be achieved are refer to the beneficial effects in the corresponding method provided above, which will not be repeated here.

[0601] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity, only the above-mentioned division of functional modules is taken as an example, and in actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0602] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, and the division of modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0603] The units described as separate components can or can not be physically separate, and the components shown as units can be one physical unit or multiple physical units, that is, they can be located in one place or distributed in multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0604] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0605] Any content of each embodiment of the present application, and any content of the same embodiment, can be freely combined. Any combination of the above is within the scope of the present application.

[0606] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, includes several instructions to make a device (which can be a single-chip microcomputer, a chip, etc.) or a processor execute all or part of the steps of the various embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0607] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are only illustrative, not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope of protection of the claims, and all of them belong to the protection of the present application.

[0608] The steps of the method or algorithm described in combination with the disclosure of the embodiments of the present application can be implemented in hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a compact disc (CD-ROM), or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0609] Those skilled in the art can understand that the functions described in the embodiments of the present application in the one or more examples above can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium. The computer readable medium includes computer readable storage medium and communication medium, wherein the communication medium includes any medium that facilitates transfer of a computer program from one place to another. The storage medium can be any available medium accessible by a general purpose or special purpose computer.

[0610] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the specific embodiments described above, which are merely illustrative rather than restrictive, and those of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims.

Claims

1. An edge-cloud collaborative system, characterized in that, The edge-cloud collaborative system includes a server and terminal devices. The server includes a first rendering module, an encoder, and a first communication module. The terminal devices include a second communication module, a second rendering module, and a decoder. The first rendering module is used to perform rendering processing on a three-dimensional scene based on rendering parameters to obtain a rendered image; wherein, the rendering parameters include first rendering parameters obtained from the terminal device; and the first rendering intermediate result is selected based on the rendering intermediate result generated during the rendering process. The encoder is configured to encode the rendered image based on the first intermediate rendering result to obtain the encoded data of the residual block and write the encoded data of the residual block into the bitstream; wherein, the residual block is obtained by performing residual calculation between the block to be encoded and the corresponding prediction block in the rendered image, and the prediction block is obtained by predicting the block to be encoded; the first intermediate rendering result is applied to at least one of the processing steps in the encoding process: block segmentation, prediction, or filtering; the bitstream does not include the encoded data of the first intermediate rendering result; The first communication module is used to send the code stream; The second communication module is used to receive the bitstream; The decoder is used to parse the bitstream to obtain a parsing result, the parsing result including the residual block corresponding to the current frame; The second rendering module is used to perform rendering processing on the three-dimensional scene based on the rendering parameters corresponding to the current frame, and generate a first intermediate rendering result during the rendering process; wherein, the rendering parameters corresponding to the current frame include the first rendering parameters generated by the terminal device; The decoder is further configured to reconstruct the image of the current frame based on the first intermediate rendering result generated by the second rendering module and the residual block corresponding to the current frame; wherein the first intermediate rendering result generated by the second rendering module is applied to at least one of the processing steps in the reconstruction process: prediction or filtering.

2. An edge-cloud collaborative system, characterized in that, The edge-cloud collaborative system includes a server and terminal devices. The server includes a first rendering module, an encoder, and a first communication module. The terminal devices include a second communication module, a second rendering module, and a decoder. The first rendering module is used to perform rendering processing on a three-dimensional scene based on rendering parameters to obtain a rendered image; wherein, the rendering parameters include first rendering parameters obtained from the terminal device; and the first rendering intermediate result is selected based on the rendering intermediate result generated during the rendering process. The encoder is configured to encode the rendered image based on the first intermediate rendering result to obtain encoded data of residual blocks and write the encoded data of the residual blocks into the bitstream; and to encode the second intermediate rendering result and write the encoded data of the second intermediate rendering result into the bitstream; wherein the residual block is obtained by performing residual calculation between the block to be encoded and the corresponding prediction block in the rendered image, and the prediction block is obtained by predicting the block to be encoded; the first intermediate rendering result is applied to at least one of the processing steps in the encoding process: block segmentation, prediction, or filtering; the second intermediate rendering result is a portion of the first intermediate rendering result; The first communication module is used to send the code stream; The second communication module is used to receive the bitstream; The decoder is used to parse the bitstream to obtain a parsing result, the parsing result including the residual block corresponding to the current frame and the second rendering intermediate result corresponding to the current frame; The second rendering module is used to render a 3D scene based on the rendering parameters corresponding to the current frame, and during the rendering process, to generate a third rendering intermediate result based on the second rendering intermediate result corresponding to the current frame; wherein, the rendering parameters corresponding to the current frame include the first rendering parameters generated by the terminal device. The decoder is further configured to perform reconstruction based on the first intermediate rendering result and the residual block corresponding to the current frame to obtain the reconstructed image of the current frame; wherein the first intermediate rendering result used to determine the reconstructed image of the current frame includes the second intermediate rendering result and the third intermediate rendering result corresponding to the current frame, and the first intermediate rendering result used to determine the reconstructed image of the current frame is applied to at least one of the processing steps in the reconstruction process: prediction or filtering.

3. An encoding method, characterized in that, Applied to a server, the method includes: The three-dimensional scene is rendered based on rendering parameters to obtain a rendered image; wherein, the rendering parameters include first rendering parameters obtained from the terminal device; Based on the intermediate rendering results generated during the rendering process, the first intermediate rendering result is selected; The rendered image is encoded based on the first intermediate rendering result to obtain the encoded data of the residual block and the encoded data of the residual block is written into the bitstream; The residual block is obtained by performing a residual operation between the block to be encoded and the corresponding prediction block in the rendered image, and the prediction block is obtained by predicting the block to be encoded. The first intermediate rendering result is applied to at least one of the processing steps in the encoding process: block segmentation, prediction, or filtering. The bitstream does not include the encoded data of the first intermediate rendering result, or the bitstream also includes the encoded data of the second intermediate rendering result, where the second intermediate rendering result is a part of the first intermediate rendering result.

4. The method according to claim 3, characterized in that, The bitstream further includes: a first indicator and / or a second indicator, wherein, The first indicator is used to indicate whether the bitstream includes the encoded data of the second rendering intermediate result, or whether it includes the encoded data of the first rendering intermediate result; The second indicator is used to indicate the type and subtype of the second rendering intermediate result, or to indicate the type of the first rendering intermediate result.

5. The method according to claim 3 or 4, characterized in that, The rendering parameters also include the second rendering parameters generated by the server; the method further includes: The third rendering parameter is written into the bitstream, wherein the third rendering parameter includes all or some of the parameters in the second rendering parameter.

6. The method according to claim 3 or 4, characterized in that, The rendering parameters also include second rendering parameters generated by the server; the bitstream also includes: a third indicator and / or a fourth indicator, wherein: The third indicator is used to indicate whether the bitstream includes a third rendering parameter, wherein the third rendering parameter includes all or some of the parameters in the second rendering parameter; The fourth indicator is used to indicate the type of the third rendering parameter.

7. The method according to claim 3, characterized in that, The first intermediate rendering result is applied to the block in the encoding process, and the block to be encoded is a prediction unit; The step of encoding the rendered image based on the first intermediate rendering result to obtain the encoded data of the residual block includes: The rendered image is divided into blocks based on the first intermediate rendering result to obtain multiple prediction units; Based on the reconstruction blocks, the multiple prediction units are predicted to obtain multiple prediction blocks, and the multiple prediction blocks correspond one-to-one with the multiple prediction units; The residual blocks between the plurality of prediction blocks and the plurality of prediction units are encoded to obtain the encoded data of the plurality of residual blocks, wherein the plurality of residual blocks correspond one-to-one with the plurality of prediction units.

8. The method according to claim 3, characterized in that, The first intermediate rendering result is applied to the prediction in the encoding process; The step of encoding the rendered image based on the first intermediate rendering result to obtain the encoded data of the residual block includes: Based on the reconstructed blocks and the first intermediate rendering result, the blocks to be encoded in the rendered image are predicted to obtain the predicted blocks corresponding to the blocks to be encoded. The residual block between the block to be encoded and the prediction block corresponding to the block to be encoded is encoded to obtain the encoded data of the residual block.

9. The method according to claim 3 or 4, characterized in that, The first intermediate rendering result is applied to the filtering in the encoding process; The step of encoding the rendered image based on the first intermediate rendering result to obtain the encoded data of the residual block includes: Based on the reconstructed blocks, the blocks to be encoded in the rendered image are predicted to obtain the predicted blocks corresponding to the blocks to be encoded. The reconstructed blocks are the filtered reconstructed blocks obtained by filtering the reconstructed blocks to be filtered based on the first intermediate rendering result. The residual block between the block to be encoded and the prediction block corresponding to the block to be encoded is encoded to obtain the encoded data of the residual block.

10. The method according to claim 7, characterized in that, The first intermediate rendering result also applies to the prediction in the encoding process; The prediction of the plurality of prediction units based on the reconstruction blocks to obtain a plurality of prediction blocks includes: Based on the reconstructed block and the first intermediate rendering result, predictions are made on the plurality of prediction units to obtain the plurality of prediction blocks.

11. The method according to claim 7, 8, or 10, characterized in that, The first intermediate rendering result also applies to the filtering in the encoding process, and the reconstructed block is the filtered reconstructed block obtained by filtering the reconstructed block to be filtered based on the first intermediate rendering result.

12. The method according to claim 7, characterized in that, The first intermediate rendering result is depth information. The step of dividing the rendered image into blocks based on the first intermediate rendering result to obtain multiple prediction units includes: The rendered image is divided into multiple coding units; Based on the depth information, computer graphics edge CGE information is generated, and the CGE information includes object edge information of objects in the rendered image; The multiple coding units are divided based on the CGE information to obtain the multiple prediction units.

13. The method according to claim 8, characterized in that, The first intermediate rendering result is a computer graphics motion vector (CGMV), which describes the displacement relationship between pixels in the rendered image and pixels in a reference frame of the rendered image. The step of predicting the block to be encoded in the rendered image based on the reconstructed block and the first intermediate rendering result to obtain the predicted block corresponding to the block to be encoded includes: From the reference frame of the rendered image, determine the reconstructed block that matches the block to be encoded; The prediction block is generated based on the CGMV and the reconstructed block that matches the block to be encoded.

14. The method according to claim 9, characterized in that, The first intermediate rendering result is a Render ID, which describes the object to which the pixel in the block to be filtered and reconstructed belongs; the method further includes: Based on the Render ID, determine the filtering parameters; The block to be reconstructed is filtered based on the filtering parameters.

15. The method according to claim 12, characterized in that, The CGE information is encoded, and the encoded data of the CGE information is written into the bitstream.

16. A decoding method, characterized in that, Applied to a terminal device, the method includes: Receive the bitstream; The bitstream is parsed to obtain a parsing result, which includes the residual block of the current frame corresponding to the 3D scene; The 3D scene is rendered based on the rendering parameters corresponding to the current frame, and a first intermediate rendering result is generated during the rendering process; wherein, the rendering parameters include the first rendering parameters generated by the terminal device; Reconstruction is performed based on the first intermediate rendering result and the residual block to obtain the reconstructed image of the current frame; wherein the first intermediate rendering result is applied to at least one of the processing steps in the reconstruction process: prediction or filtering.

17. The method according to claim 16, characterized in that, The parsing result further includes: a first indicator and a second indicator, wherein the first indicator is used to indicate whether the bitstream includes encoded data of a first rendering intermediate result generated by the server, and the second indicator is used to indicate the type of the first rendering intermediate result generated by the server; The generation of the first intermediate rendering result during the rendering process includes: When it is determined based on the first indicator that the bitstream does not include the encoded data of the first intermediate rendering result generated by the server, the first intermediate rendering result is generated during the rendering process based on the second indicator.

18. The method according to claim 16 or 17, characterized in that, The parsing result also includes: a third indicator and a fourth indicator; the third indicator is used to indicate whether the bitstream includes a third rendering parameter, and the fourth indicator is used to indicate the type of the third rendering parameter; The method further includes: When it is determined that the bitstream includes the third rendering parameter based on the third indicator, the type of the third rendering parameter is determined based on the fourth indicator. Based on the type of the third rendering parameter, other types of fourth rendering parameters are generated; the other types are types other than the type of the third rendering parameter among the various types corresponding to the second rendering parameter generated by the server. The rendering parameters also include the third rendering parameter and the fourth rendering parameter.

19. The method according to claim 18, characterized in that, The method further includes: When it is determined based on the third indicator that the bitstream does not include the third rendering parameter, the terminal device generates the second rendering parameter corresponding to the current frame; The rendering parameters also include second rendering parameters generated by the terminal device.

20. The method according to claim 16, characterized in that, The residual blocks are multiple, and the first intermediate rendering result is used for prediction in the reconstruction process; The step of reconstructing based on the first intermediate rendering result and the residual block to obtain the reconstructed image of the current frame includes: Based on the first intermediate rendering result, determine the block information of the current frame; Based on the block information and reconstruction block, multiple prediction units included in the current frame are predicted to obtain multiple prediction blocks. The multiple prediction blocks correspond one-to-one with the multiple prediction units, and the multiple residual blocks correspond one-to-one with the multiple prediction units. The reconstructed image of the current frame is determined based on the plurality of residual blocks and the plurality of prediction blocks.

21. The method according to claim 16, characterized in that, The first intermediate rendering result is used for prediction in the reconstruction process; The step of reconstructing based on the first intermediate rendering result and the residual block to obtain the reconstructed image of the current frame includes: Based on the first rendering intermediate result and the reconstructed block, the block to be decoded in the current frame is predicted to obtain the predicted block corresponding to the block to be decoded; The reconstructed image of the current frame is determined based on the residual block corresponding to the block to be decoded and the prediction block corresponding to the block to be decoded in the residual block of the current frame.

22. The method according to claim 16, 17, or 19, characterized in that, The first intermediate rendering result is applied to the filtering in the reconstruction process; the reconstruction based on the first intermediate rendering result and the residual block to obtain the reconstructed image of the current frame includes: Based on the reconstructed block, the block to be decoded in the current frame is predicted to obtain the predicted block corresponding to the block to be decoded. The reconstructed block is a filtered reconstructed block obtained by filtering the reconstructed block to be filtered based on the first intermediate rendering result. The reconstructed image of the current frame is determined based on the residual block corresponding to the block to be decoded and the prediction block corresponding to the block to be decoded in the residual block of the current frame.

23. The method according to claim 20, characterized in that, Based on the block information and reconstruction blocks, prediction is performed on multiple prediction units included in the current frame to obtain multiple prediction blocks, including: Based on the block information, the first rendering intermediate result, and the reconstruction block, multiple prediction units included in the current frame are predicted to obtain multiple prediction blocks.

24. The method according to claim 20, 21, or 23, characterized in that, The first intermediate rendering result also applies to the filtering in the reconstruction process, and the reconstructed block is the filtered reconstructed block obtained by filtering the reconstructed block to be filtered based on the first intermediate rendering result.

25. The method according to claim 20 or 23, characterized in that, The first intermediate rendering result is depth information. Determining the block information of the current frame based on the first intermediate rendering result includes: Based on the depth information, computer graphics edge CGE information is generated, and the CGE information includes object edge information of objects in the current frame; Based on the CGE information, the block information of the current frame is determined.

26. The method according to claim 21, characterized in that, The first intermediate rendering result is a computer graphics motion vector (CGMV), which describes the displacement relationship between pixels in the current frame and pixels in a reference frame. The step of predicting the block to be decoded in the current frame based on the first intermediate rendering result and the reconstructed block to obtain the predicted block corresponding to the block to be decoded includes: From the reference frames of the current frame, determine the reconstructed block that matches the block to be decoded; Based on the CGMV and the reconstructed block that matches the block to be decoded, a prediction block corresponding to the block to be decoded is generated.

27. The method according to claim 24, characterized in that, The first intermediate rendering result is a Render ID, which describes the object to which the pixel in the block to be filtered and reconstructed belongs; the method further includes: Based on the Render ID, determine the filtering parameters; The block to be reconstructed is filtered based on the filtering parameters.

28. The method according to claim 16, characterized in that, The residual blocks are multiple, and the first intermediate rendering result is used for prediction in the reconstruction process; the parsing result also includes CGE information, which includes object edge information of objects in the current frame; The step of reconstructing based on the first intermediate rendering result and the residual block to obtain the reconstructed image of the current frame includes: Based on the CGE information, determine the block information of the current frame; Based on the block information and reconstruction block, multiple prediction units included in the current frame are predicted to obtain multiple prediction blocks. The multiple prediction blocks correspond one-to-one with the multiple prediction units, and the multiple residual blocks correspond one-to-one with the multiple prediction units. The reconstructed image of the current frame is determined based on the plurality of residual blocks and the plurality of prediction blocks.

29. A decoding method, characterized in that, The method is applied to a terminal device, and the method includes: Receive the bitstream; The bitstream is parsed to obtain the parsing result, which includes: the residual block of the current frame corresponding to the 3D scene and the second rendering intermediate result corresponding to the current frame; The three-dimensional scene is rendered based on the rendering parameters corresponding to the current frame, and a third rendering intermediate result is generated based on the second rendering intermediate result during the rendering process. The rendering parameters include the first rendering parameters generated by the terminal device. Reconstruction is performed based on the first intermediate rendering result and the residual block to obtain the reconstructed image of the current frame; wherein, the first intermediate rendering result includes the second intermediate rendering result and the third intermediate rendering result, and the first intermediate rendering result is applied to at least one of the processing steps in the reconstruction process: prediction or filtering.

30. The method according to claim 29, characterized in that, The parsing result further includes: a second indicator, which is used to indicate the type and subtype of the second rendering intermediate result; The step of generating a third intermediate rendering result based on the second intermediate rendering result during the rendering process includes: Based on the second indicator, determine the type and subtype corresponding to the second rendering intermediate result; During the rendering process, based on the type and subtype corresponding to the second intermediate rendering result, a third intermediate rendering result of other subtypes is generated; wherein, the other subtypes are the subtypes included in the type corresponding to the second intermediate rendering result, excluding the subtype corresponding to the second intermediate rendering result.

31. The method according to claim 29 or 30, characterized in that, The parsing result also includes: a third indicator and a fourth indicator; the third indicator is used to indicate whether the bitstream includes a third rendering parameter, and the fourth indicator is used to indicate the type of the third rendering parameter; The method further includes: When it is determined that the bitstream includes the third rendering parameter based on the third indicator, the type of the third rendering parameter is determined based on the fourth indicator. Based on the type of the third rendering parameter, other types of fourth rendering parameters are generated; the other types are types other than the type of the third rendering parameter among the various types corresponding to the second rendering parameter generated by the server. The rendering parameters also include the third rendering parameter and the fourth rendering parameter.

32. The method according to claim 31, characterized in that, The method further includes: When it is determined based on the third indicator that the bitstream does not include the third rendering parameter, the terminal device generates the second rendering parameter corresponding to the current frame; The rendering parameters also include second rendering parameters generated by the terminal device.

33. The method according to claim 29, characterized in that, The residual blocks are multiple, and the first intermediate rendering result is used for prediction in the reconstruction process; The step of reconstructing based on the first intermediate rendering result and the residual block to obtain the reconstructed image of the current frame includes: Based on the first intermediate rendering result, determine the block information of the current frame; Based on the block information and reconstruction block, multiple prediction units included in the current frame are predicted to obtain multiple prediction blocks. The multiple prediction blocks correspond one-to-one with the multiple prediction units, and the multiple residual blocks correspond one-to-one with the multiple prediction units. The reconstructed image of the current frame is determined based on the plurality of residual blocks and the plurality of prediction blocks.

34. The method according to claim 29, characterized in that, The first intermediate rendering result is used for prediction in the reconstruction process; The step of reconstructing based on the first intermediate rendering result and the residual block to obtain the reconstructed image of the current frame includes: Based on the first rendering intermediate result and the reconstructed block, the block to be decoded in the current frame is predicted to obtain the predicted block corresponding to the block to be decoded; The reconstructed image of the current frame is determined based on the residual block corresponding to the block to be decoded and the prediction block corresponding to the block to be decoded in the residual block of the current frame.

35. The method according to any one of claims 29, 30, or 32, characterized in that, The first intermediate rendering result is applied to the filtering in the reconstruction process; the reconstruction based on the first intermediate rendering result and the residual block to obtain the reconstructed image of the current frame includes: Based on the reconstructed block, the block to be decoded in the current frame is predicted to obtain the predicted block corresponding to the block to be decoded. The reconstructed block is a filtered reconstructed block obtained by filtering the reconstructed block to be filtered based on the first intermediate rendering result. The reconstructed image of the current frame is determined based on the residual block corresponding to the block to be decoded and the prediction block corresponding to the block to be decoded in the residual block of the current frame.

36. The method according to claim 33, characterized in that, Based on the block information and reconstruction blocks, prediction is performed on multiple prediction units included in the current frame to obtain multiple prediction blocks, including: Based on the block information, the first rendering intermediate result, and the reconstruction block, multiple prediction units included in the current frame are predicted to obtain multiple prediction blocks.

37. The method according to claim 33, 34, or 36, characterized in that, The first intermediate rendering result also applies to the filtering in the reconstruction process; the reconstructed block is a filtered reconstructed block obtained by filtering the reconstructed block to be filtered based on the first intermediate rendering result.

38. The method according to claim 36, characterized in that, The first intermediate rendering result is depth information. Determining the block information of the current frame based on the first intermediate rendering result includes: Based on the depth information, computer graphics edge CGE information is generated, and the CGE information includes object edge information of objects in the current frame; Based on the CGE information, the block information of the current frame is determined.

39. The method according to claim 34, characterized in that, The first intermediate rendering result is a computer graphics motion vector (CGMV), which describes the displacement relationship between pixels in the current frame and pixels in a reference frame. The step of predicting the block to be decoded in the current frame based on the first intermediate rendering result and the reconstructed block to obtain the predicted block corresponding to the block to be decoded includes: From the reference frames of the current frame, determine the reconstructed block that matches the block to be decoded; Based on the CGMV and the reconstructed block that matches the block to be decoded, a prediction block corresponding to the block to be decoded is generated.

40. The method according to claim 37, characterized in that, The first intermediate rendering result is a Render ID, which describes the object to which the pixel in the block to be filtered and reconstructed belongs; the method further includes: Based on the Render ID, determine the filtering parameters; The block to be reconstructed is filtered based on the filtering parameters.

41. The method according to claim 29, characterized in that, The residual blocks are multiple, and the first intermediate rendering result is used for prediction in the reconstruction process; the parsing result also includes CGE information, which includes object edge information of objects in the current frame; The step of reconstructing based on the first intermediate rendering result and the residual block to obtain the reconstructed image of the current frame includes: Based on the CGE information, determine the block information of the current frame; Based on the block information and reconstruction block, multiple prediction units included in the current frame are predicted to obtain multiple prediction blocks. The multiple prediction blocks correspond one-to-one with the multiple prediction units, and the multiple residual blocks correspond one-to-one with the multiple prediction units. The reconstructed image of the current frame is determined based on the plurality of residual blocks and the plurality of prediction blocks.

42. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is coupled to the processor; The memory stores program instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 3 to 41.

43. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed on a computer or processor, causes the computer or processor to perform the method of any one of claims 3 to 41.

44. A computer program product, characterized in that, The computer program product includes a software program that, when executed by a computer or processor, causes the steps of the method according to any one of claims 3 to 41 to be performed.

Images