Device-cloud collaboration system, encoding and decoding method, and electronic device

Abstract

Embodiments of the present application provide a device-cloud collaboration system, an encoding and decoding method, and an electronic device. The encoding method includes: performing a rendering process on a three-dimensional scene based on rendering parameters to obtain a rendered image, the rendering parameters including first rendering parameters obtained from a terminal device; selecting a first intermediate rendering result based on intermediate rendering results generated in the rendering process; generating a virtual reference frame based on the first intermediate rendering result; predicting the rendered image based on the virtual reference frame to obtain a predicted image; encoding a residual image between the predicted image and the rendered image; and encoding encoded data of the residual image into a bitstream. The bitstream does not include the encoded data of the first intermediate rendering result. In this way, the server does not need to transmit the intermediate rendering result to the terminal device, thereby reducing interaction delay while ensuring that the bitrate overhead of the data stream transmitted by the server to the terminal device is effectively reduced.

Description

[Technical Field]

[0001] This application claims priority to Chinese Patent Application No. 202211703245.1, entitled "Device-Cloud Collaboration System, Encoding and Decoding Method, and Electronic Device," filed with the State Intellectual Property Administration of China on December 29, 2022, which is incorporated herein by reference in its entirety.

[0002] TECHNICAL FIELD Embodiments of the present application relate to the field of encoding and decoding, and in particular to a device-cloud collaboration system, an encoding and decoding method, and an electronic device. [Background technology]

[0003] In many situations (e.g., games, virtual reality (VR) / augmented reality (AR)), rendering needs to be performed to generate images in order to make the resulting images more realistic and improve the user experience. Rendering requires powerful computing power. Because it is limited by physical conditions of the target, such as device size and power consumption, the computing power of the device-side device is usually much weaker than that of the cloud-side server. Therefore, rendering is usually deployed on the cloud-side server. The cloud-side server performs rendering, compresses the rendered images / videos, and sends the compressed rendered images / videos to the device-side device for display by the device-side device.

[0004] As people's demands for rendering quality continue to improve and the resolution of display devices continues to improve, the image quality and resolution of rendered images / videos also continue to improve. As a result, the bitrate overhead of compressed rendered images / videos increases, network bandwidth occupation increases, and interaction delays become longer. In conventional technologies, a cloud-side server typically encodes and transmits rendered low-resolution images / videos, and then transmits intermediate rendering results generated in the process of rendering high-resolution images / videos to a device-side device. Based on the intermediate rendering results delivered by the cloud-side server, the device-side device performs upsampling on the rendered low-resolution images / videos delivered by the cloud side to generate high-resolution images / videos for display. In this way, the bitrate overhead can be reduced to a certain extent, but the encoding efficiency remains low. Summary of the Invention

[0005] In light of this, the present application provides a device-cloud collaboration system, an encoding and decoding method, and an electronic device. The encoding and decoding method is implemented based on the device-cloud collaboration system, and can reduce interaction delay while ensuring that the bit rate overhead of the data stream sent by the server to the terminal device is effectively reduced. [Means for solving the problem]

[0006] According to a first aspect, an embodiment of the present application provides a device-cloud collaboration system, the device-cloud collaboration system including a server and a terminal device, the server including a first rendering module, an encoder, and a first communication module, and the terminal device including a second communication module, a second rendering module, and a decoder.

[0007] The first rendering module is configured to perform a rendering process on the three-dimensional scene based on rendering parameters to obtain a rendered image, the rendering parameters including first rendering parameters obtained from the terminal device, select a first intermediate rendering result based on intermediate rendering results generated in the rendering process, and generate a virtual reference frame based on the first intermediate rendering result.

[0008] The encoder is configured to predict the rendered image based on the virtual reference frame to obtain a predicted image, encode a residual image between the predicted image and the rendered image, and encode the encoded data of the residual image into a bitstream, wherein the bitstream does not include the encoded data of the first intermediate rendering result.

[0009] The first communication module is configured to transmit the bitstream.

[0010] The second communication module is configured to receive the bitstream.

[0011] The decoder is configured to parse the bitstream to obtain an analysis result, the analysis result including a residual image corresponding to the current frame.

[0012] The second rendering module is configured to perform a rendering process on the three-dimensional scene based on rendering parameters corresponding to a current frame, and generate a first intermediate rendering result in the rendering process, where the rendering parameters corresponding to the current frame include the first rendering parameters generated by the terminal device, and to generate a virtual reference frame based on the first intermediate rendering result generated by the second rendering module.

[0013] The decoder is further configured to predict the current frame based on the virtual reference frame generated by the second rendering module to obtain a predicted image, and to perform reconstruction based on the predicted image determined by the decoder and a residual image corresponding to the current frame to obtain a reconstructed image of the current frame.

[0014] In this way, all rendering is performed by the terminal device, and the server does not need to send intermediate rendering results to the terminal device. Therefore, the present application can reduce interaction delay while ensuring that the bit rate overhead of the data stream sent by the server to the terminal device is effectively reduced. In addition, there is a strong correlation between the intermediate rendering results and the rendered image. Therefore, the present application can guarantee the image reconstruction quality because the rendered image is encoded based on the intermediate rendering results.

[0015] For example, the server may be a game server, the server may be a single server, or may be a server cluster, which is not limited in this application.

[0016] For example, terminal devices include, but are not limited to, personal computers, computer workstations, smartphones, tablet computers, servers, smart cameras, intelligent vehicles, other types of mobile phones, media consumption devices, wearable devices (e.g., VR / AR helmets or VR glasses), set-top boxes, game consoles, etc.

[0017] For example, the rendering parameters may be all parameters that are input to a graphics rendering engine and are required for rendering processing by the graphics rendering engine, and may include various parameters used in rendering, position vectors and color vectors of all light sources, position vectors of the player or observer, information such as the sampling method of each texture and the position coordinates of objects in each scene, motion tracks of moving objects, skeletal animation parameters, etc. This is not limited in the present application.

[0018] For example, the intermediate rendering result may be intermediate data used to generate the image / video to be displayed and generated by a graphics rendering engine in the process of generating the image (i.e., the rendered image) / video (i.e., the rendered video) to be displayed. For example, the intermediate rendering result may include, but is not limited to, a computer graphics motion vector (CGMV), an intermediate rendered image (an intermediate rendered image is an image generated before a final rendered image (i.e., the aforementioned rendered image) is generated, and the calculation complexity of the intermediate rendered image is lower than that of a rendered image, and the intermediate rendered image may be, for example, an intermediate rendered image on which indirect lighting rendering has not been performed, an intermediate rendered image on which specular reflection processing has not been performed, or an intermediate rendered image on which highlight processing has not been performed), a position map, a normal map, an albedo map, a specular intensity map, a mesh identifier (Mesh ID), a material ID (each material map corresponds to one material ID), a render ID (each object (or one three-dimensional object model) corresponds to one Render ID), depth information, etc. This is not limited in the present application. The first intermediate rendering result is a part of all intermediate rendering results generated in the rendering process. It should be noted that the type of intermediate result included in the first intermediate rendering result generated by the terminal device is the same as the type of intermediate result included in the first intermediate rendering result generated by the server, and the accuracy of the intermediate result included in the first intermediate rendering result generated by the terminal device is less than or equal to the accuracy of the intermediate result included in the first intermediate rendering result generated by the server.

[0019] It should be understood that if the server performs lossy coding on the residual image, the residual image obtained by the terminal device through analysis will be different from the residual image encoded by the server, whereas if the server performs lossless coding on the residual block, the residual image obtained by the terminal device through analysis will be the same as the residual image encoded by the server.

[0020] For example, a virtual reference frame is a reference frame generated based on prior information, which includes information other than the decoded information in the encoder and the video / image to be encoded. To further remove video temporal redundancy, a virtual reference frame may be used as supplementary information for inter-coding.

[0021] It should be understood that the server in this application may include more or fewer modules than those described above, and this is not a limitation of this application. The terminal device in this application may include more or fewer modules than those described above, and this is not a limitation of this application.

[0022] It should be understood that the video coding standard used by the encoder and decoder is not limited in this application. For example, the video coding standard may include, but is not limited to, H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), H.266 / VVC (Versatile Video Coding), AV1 (AOMedia Video 1, "AOMedia" is video coding developed by the Alliance for Open Media), etc., and extension standards of these video coding standards. In addition, the video coding standard may further include new video coding standards and extension standards created with the development of video coding and decoding technologies.

[0023] According to a second aspect, an embodiment of the present application provides an encoding method applied to a server, the method including: performing a rendering process on a three-dimensional scene based on rendering parameters to obtain a rendered image, the rendering parameters including first rendering parameters obtained from a terminal device; selecting a first intermediate rendering result based on intermediate rendering results generated in the rendering process; generating a virtual reference frame based on the first intermediate rendering result; predicting the rendered image based on the virtual reference frame to obtain a predicted image; encoding a residual image between the predicted image and the rendered image, and encoding encoded data of the residual image into a bitstream, where the bitstream does not include the encoded data of the first intermediate rendering result.

[0024] In this way, all rendering is performed by the terminal device, and the server does not need to send intermediate rendering results to the terminal device. Therefore, the present application can reduce interaction delay while ensuring that the bit rate overhead of the data stream sent by the server to the terminal device is effectively reduced. In addition, there is a strong correlation between the intermediate rendering results and the rendered image. Therefore, the present application can guarantee the image reconstruction quality because the rendered image is encoded based on the intermediate rendering results.

[0025] In one possible approach, a virtual reference frame may be used as a reference frame for the rendered image, and then a predicted block (i.e., a predicted image) that matches the block to be coded in the rendered image is searched for from the virtual reference frame.

[0026] In one possible scheme, both the virtual reference frame and the original reference frame may be used as candidate reference frames for the rendered image. The original reference frame is a reconstructed image. For a block to be coded in the rendered image, inter prediction may be performed based on multiple candidate reference frames, and multiple prediction blocks may be determined. One candidate reference frame corresponds to one prediction block. An optimal prediction block may be selected from the multiple prediction blocks (e.g., using rate-distortion cost as an evaluation criterion, the prediction block with the smallest rate-distortion cost may be determined as the optimal prediction block).

[0027] For example, the first intermediate rendering result is part of the intermediate rendering results generated in the rendering process.

[0028] According to a second aspect, the bitstream further includes a first instruction identifier and / or a second instruction identifier. The first instruction identifier indicates whether the bitstream includes encoded data of a first intermediate rendering result, and the second instruction identifier indicates the type of the first intermediate rendering result. In this way, the terminal device knows whether the bitstream includes the first intermediate rendering result and the specific type of the first intermediate rendering result to be generated.

[0029] For example, the first intermediate rendering result may be classified into multiple types, such as a motion vector type and an image type. If the first intermediate rendering result is a CGMV, the corresponding type may be a motion vector type. If the first intermediate rendering result is an intermediate rendered image, the corresponding type may be an image type. It should be understood that the first intermediate rendering result may further include another type, which is not limited in this application.

[0030] In one possible approach, the intermediate rendered image may be an image generated by performing some rendering work with a graphics rendering engine (the rendered image is an image generated by performing all rendering work with the server's graphics rendering engine).

[0031] In one possible approach, the intermediate rendered image may be an image generated by performing rendering by a graphics rendering engine of the server based on low-precision rendering parameters or some rendering parameters (the rendered image is an image generated by performing rendering by a graphics rendering engine based on all high-precision rendering parameters).

[0032] According to the second aspect or any one of the implementations of the second aspect, the rendering parameters further include second rendering parameters generated by the server. The method further includes encoding third rendering parameters into the bitstream. The third rendering parameters include all or part of the parameters of the second rendering parameters.

[0033] Because the rendering parameters generated by the server are more accurate than the rendering parameters generated by the terminal device, the server may transmit some or all of the second rendering parameters to the terminal device, so that the first intermediate rendering result generated by the terminal device can be more accurate, the accuracy of the virtual reference frame can be improved, and the quality of the image obtained by decoding based on the virtual reference frame can be improved.

[0034] In addition, the data volume of the second rendering parameters is small (several / tens of KB), which is much smaller than the data volume of the intermediate rendering results. Therefore, even if the rendering parameters are transmitted to the terminal device in the present application, the bit rate overhead of the data stream transmitted by the server to the terminal device in the present application is smaller than the bit rate overhead of the data stream transmitted by the server to the terminal device in the prior art. In addition, the computing power of the terminal device can be further saved.

[0035] It should be noted that the first rendering parameter and the second rendering parameter may form the rendering parameters (i.e., all parameters input to the graphics rendering engine and required for the rendering process by the graphics rendering engine).

[0036] For example, the third rendering parameter may be encoded, and the encoded data of the third rendering parameter may be encoded into the bitstream, or the third rendering parameter may be directly added to the bitstream without being encoded, which is not limited in this application.

[0037] According to the second aspect or any one of the implementations of the second aspect, the rendering parameters further include a second rendering parameter generated by the server. The bitstream further includes a third instruction identifier and / or a fourth instruction identifier. The third instruction identifier indicates whether the bitstream includes the third rendering parameter. The third rendering parameter includes all or part of the parameters of the second rendering parameter. The fourth instruction identifier indicates the type of the third rendering parameter. In this way, the terminal device knows whether the bitstream includes the third rendering parameter. If the third rendering parameter is part of the second rendering parameter, the terminal device can generate a fourth rendering parameter based on the type of the third rendering parameter. The fourth rendering parameter is part of the second rendering parameter other than the third rendering parameter.

[0038] For example, the second rendering parameters may be classified into multiple types, such as type C1 and type C2. For example, the second rendering parameters may include motion information of a rigid motion object and motion information of a non-rigid dynamic object. The type corresponding to the motion information of the rigid motion object is type C1, and the type corresponding to the motion information of the non-rigid dynamic object is type C2.

[0039] In one possible way, the third rendering parameters may include motion information of rigid motion objects and motion information of non-rigid dynamic objects.

[0040] In one possible manner, the third rendering parameter may include motion information of a rigid motion object. In this way, compared with the third rendering parameter including the motion information of a rigid motion object and the motion information of a non-rigid dynamic object, the third rendering parameter including the motion information of a rigid dynamic object can further reduce the bit rate overhead of the data stream transmitted by the server to the terminal device.

[0041] In one possible manner, the third rendering parameter may include motion information of a non-rigid dynamic object. In this way, compared with the third rendering parameter including the motion information of a rigid motion object and the motion information of a non-rigid dynamic object, the third rendering parameter including the motion information of a non-rigid dynamic object can further reduce the bit rate overhead of the data stream transmitted by the server to the terminal device.

[0042] According to the second aspect or any one of the implementations of the second aspect, the step of generating a virtual reference frame based on the first intermediate rendering result includes the steps of determining a type of the first intermediate rendering result and generating a virtual reference frame based on the first intermediate rendering result and the type of the first intermediate rendering result.

[0043] According to the second aspect or any one of the implementations of the second aspect, the step of generating a virtual reference frame based on the first intermediate rendering result and a type of the first intermediate rendering result includes, when the type of the first intermediate rendering result is a motion vector type, generating the virtual reference frame based on the first intermediate rendering result and the reconstructed image, where the first intermediate rendering result is a computer graphics motion vector CGMV, and the CGMV is used to describe a displacement relationship between samples in the rendered image and samples in the reconstructed image.

[0044] To avoid inaccurate motion estimation in the prior art, CGMV is generated by graphic means. In addition, CGMV is a pixel-level MV, while the MV generated by existing motion estimation is an image block-level MV. To reduce prediction errors, pixel-level MV can more accurately describe the edges of objects. Therefore, by determining a virtual reference frame based on CGMV and then performing inter prediction based on the virtual reference frame, the error of the predicted block can be reduced to a certain extent, the accuracy of the predicted block can be improved, and the inter coding and compression efficiency can be further improved.

[0045] According to the second aspect or any one of the implementations of the second aspect, the step of generating a virtual reference frame based on the first intermediate rendering result and a type of the first intermediate rendering result includes a step of determining the intermediate rendering result as the virtual reference frame when the type of the first intermediate rendering result is determined to be an image type, the first intermediate rendering result is an intermediate rendered image, and a computational complexity corresponding to the intermediate rendered image is lower than a computational complexity corresponding to the rendered image.

[0046] The small difference between the intermediate rendered image and the rendered image allows for a small residual, which in turn allows for a reduced bit rate for the coding data of the residual block.

[0047] According to the second aspect or any one of the implementations of the second aspect, the first intermediate rendering result includes a CGMV and / or an intermediate rendered image, and the computational complexity corresponding to the intermediate rendered image is lower than the computational complexity corresponding to the rendered image.

[0048] It should be understood that the first intermediate rendering result may further include another intermediate result, which is not limited in this application.

[0049] According to a third aspect, an embodiment of the present application provides a decoding method, the decoding method including: receiving a bitstream; analyzing the bitstream to obtain an analysis result, where the analysis result includes a residual image corresponding to a current frame; performing a rendering process on a three-dimensional scene based on rendering parameters corresponding to the current frame to generate a first intermediate rendering result in the rendering process, where the rendering parameters include first rendering parameters generated by a terminal device; generating a virtual reference frame based on the first intermediate rendering result; predicting the current frame based on the virtual reference frame to obtain a predicted image; and performing reconstruction based on the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0050] According to a third aspect, the analysis result further includes a first instruction identifier and a second instruction identifier, where the first instruction identifier indicates whether the bitstream includes encoded data of a first intermediate rendering result generated by the server, and the second instruction identifier indicates a type of the first intermediate rendering result generated by the server. The generating of the first intermediate rendering result in the rendering process includes generating the first intermediate rendering result in the rendering process based on the second instruction identifier if it is determined based on the first instruction identifier that the bitstream does not include encoded data of the first intermediate rendering result generated by the server. In this way, the terminal device can generate a first intermediate rendering result whose type is the same as the type of the first intermediate rendering result generated by the server.

[0051] According to the third aspect or any one of the implementations of the third aspect, the analysis result further includes a third instruction identifier and a fourth instruction identifier, where the third instruction identifier indicates whether the bitstream includes a third rendering parameter, and the fourth instruction identifier indicates a type of the third rendering parameter. If it is determined that the bitstream includes the third rendering parameter based on the third instruction identifier, the method further includes determining a type of the third rendering parameter based on the fourth instruction identifier and generating a fourth rendering parameter of another type based on the type of the third rendering parameter. The another type is one type other than the type of the third rendering parameter among a plurality of types corresponding to the second rendering parameter generated by the server. The rendering parameters further include the third rendering parameter and the fourth rendering parameter.

[0052] In this way, if the third rendering parameter sent by the server to the terminal device is part of the second rendering parameters, the terminal device can generate rendering parameters other than the third rendering parameter among the second rendering parameters based on the fourth instruction identifier in the bitstream to obtain all rendering parameters required for the rendering process by the graphics rendering engine.

[0053] For example, if it is determined based on the fourth instruction identifier that the third rendering parameter is all of the second rendering parameter, the rendering parameters further include the third rendering parameter.

[0054] It should be understood that if the server performs lossy encoding on the third rendering parameters, the third rendering parameters obtained by the terminal device through analysis will be different from the third rendering parameters encoded by the server, and if the server performs lossless encoding on the third rendering parameters, the third rendering parameters obtained by the terminal device through analysis will be the same as the third rendering parameters encoded by the server.

[0055] According to a third aspect or any one of the implementations of the third aspect, the method further includes, when it is determined based on the third instruction identifier that the bitstream does not include the third rendering parameter, generating, by the terminal device, second rendering parameters corresponding to the current frame, wherein the rendering parameters further include the second rendering parameters generated by the terminal device.

[0056] In this way, if the server does not send the second rendering parameters to the terminal device, the terminal device can further generate the second rendering parameters in order to obtain all the rendering parameters required for the rendering process by the graphics rendering engine.

[0057] It should be understood that the parameter type included in the second rendering parameters generated by the terminal device is the same as the parameter type included in the second rendering parameters generated by the server, and the precision of the parameters included in the second rendering parameters generated by the terminal device is less than or equal to the precision of the parameters included in the second rendering parameters generated by the server.

[0058] According to the third aspect or any one of the implementations of the third aspect, the step of generating a virtual reference frame based on the first intermediate rendering result includes the steps of determining a type of the first intermediate rendering result and generating a virtual reference frame based on the first intermediate rendering result and the type of the first intermediate rendering result.

[0059] According to the third aspect or any one of the implementations of the third aspect, the step of generating a virtual reference frame based on the first intermediate rendering result and the type information of the first intermediate rendering result includes, when the type of the first intermediate rendering result is a motion vector type, generating the virtual reference frame based on the first intermediate rendering result and a reconstructed image of another frame, wherein the first intermediate rendering result is a computer graphics motion vector CGMV, and the CGMV is used to describe a displacement relationship between samples in the reconstructed image of the current frame and samples in the reconstructed image of another frame.

[0060] According to the third aspect or any one of the implementations of the third aspect, the step of generating a virtual reference frame based on the first intermediate rendering result and type information of the first intermediate rendering result includes a step of determining the first intermediate rendering result as the virtual reference frame when the type of the first intermediate rendering result is determined to be an image type, the first intermediate rendering result is an intermediate rendered image, and a computational complexity corresponding to the intermediate rendered image is lower than a computational complexity corresponding to the rendered image.

[0061] According to the third aspect or any one of the implementations of the third aspect, the first intermediate rendering result includes CGMV and / or an intermediate rendered image, and the computational complexity corresponding to the intermediate rendered image is lower than the computational complexity corresponding to the rendered image.

[0062] The third aspect and any one of the implementations of the third aspect correspond to the second aspect and any one of the implementations of the second aspect. For technical effects corresponding to the third aspect and any one of the implementations of the third aspect, please refer to the technical effects corresponding to the second aspect and any one of the implementations of the second aspect. Details will not be described again here.

[0063] According to a fourth aspect, the present application further provides a device-cloud collaboration system, which includes a server and a terminal device, wherein the server includes a first rendering module, an encoder, and a first communication module, and the terminal device includes a second communication module, a second rendering module, and a decoder.

[0064] The first rendering module is configured to perform a rendering process on the three-dimensional scene based on rendering parameters to obtain a rendered image, the rendering parameters including first rendering parameters obtained from the terminal device, select a first intermediate rendering result based on intermediate rendering results generated in the rendering process, and generate a virtual reference frame based on the first intermediate rendering result.

[0065] The encoder is configured to predict the rendered image based on the virtual reference frame to obtain a predicted image, encode a residual image between the predicted image and the rendered image, encode encoded data of the residual image into a bitstream, encode a second intermediate rendering result, and encode encoded data of the second intermediate rendering result into a bitstream, where the second intermediate rendering result is a part of the first intermediate rendering result.

[0066] The first communication module is configured to transmit the bitstream.

[0067] The second communication module is configured to receive the bitstream.

[0068] The decoder is configured to parse the bitstream to obtain an analysis result, the analysis result including a residual image corresponding to the current frame and a second intermediate rendering result corresponding to the current frame.

[0069] The second rendering module is configured to perform a rendering process on the three-dimensional scene based on rendering parameters corresponding to a current frame, generate a third intermediate rendering result based on the second intermediate rendering result corresponding to the current frame in the rendering process, the rendering parameters corresponding to the current frame including the first rendering parameters generated by the terminal device, and generate a virtual reference frame based on the first intermediate rendering result, where the first intermediate rendering result used by the second rendering module to generate the virtual reference frame includes the second intermediate rendering result and the third intermediate rendering result corresponding to the current frame.

[0070] The decoder is further configured to predict the current frame based on the virtual reference frame generated by the second rendering module to obtain a predicted image, and to perform reconstruction based on the predicted image determined by the decoder and a residual image corresponding to the current frame to obtain a reconstructed image of the current frame.

[0071] In this way, partial rendering is performed by the terminal device, and the server only needs to send some intermediate rendering results to the terminal device. Therefore, the present application can reduce interaction delay while ensuring that the bit rate overhead of the data stream sent by the server to the terminal device is effectively reduced. In addition, there is a strong correlation between the intermediate rendering results and the rendered image. Therefore, the present application can guarantee the image reconstruction quality because the rendered image is encoded based on the intermediate rendering results.

[0072] For example, assuming that the first intermediate rendering result is a CGMV, the CGMV may include a CGMV of a static object, a CGMV of a rigid dynamic object, and a CGMV of a non-rigid dynamic object, and the second intermediate rendering result may include any one or two of the CGMV of the static object, the CGMV of the rigid dynamic object, and the CGMV of the non-rigid dynamic object.

[0073] In one possible way, the server can encode the CGMV of static objects, in this way the terminal device does not need to calculate the CGMV of static objects, thus saving the computational power of the terminal device.

[0074] In one possible manner, the server can encode the CGMV of the dynamic object. In this way, the terminal device does not need to calculate the CGMV of the dynamic object, thereby saving the computational power of the terminal device. In addition, the computational cost of calculating the CGMV of the dynamic object is greater than the computational cost of calculating the CGMV of the static object. Therefore, in order to further save the computational power of the terminal device, the CGMV of the dynamic object is encoded into a second bitstream and transmitted to the terminal device.

[0075] For example, the CGMV of a dynamic object may include the CGMV of a rigid dynamic object and / or the CGMV of a non-rigid dynamic object.

[0076] In one possible scheme, both the CGMV of rigid dynamic objects and the CGMV of non-rigid dynamic objects may be coded.

[0077] In one possible scheme, the CGMV of rigid dynamic objects can be coded, thus reducing the bit rate compared to coding the CGMV of rigid and non-rigid dynamic objects.

[0078] In one possible method, the CGMV of a non-rigid dynamic object can be coded. In this way, the bit rate can be reduced compared to coding the CGMV of a rigid dynamic object and the CGMV of a non-rigid dynamic object. In addition, since the number of triangular meshes of a non-rigid dynamic object is large, the calculation amount of the CGMV of a non-rigid dynamic object is large. Therefore, compared with coding the CGMV of a rigid dynamic object, coding the CGMV of a non-rigid dynamic object can save more computational power of the terminal device.

[0079] For example, assume that the first intermediate rendering result is an intermediate rendered image, and the server can use a portion of an area image within the intermediate rendered image as the second intermediate rendering result.

[0080] According to a fifth aspect, an embodiment of the present application further provides an encoding method, the encoding method including: performing a rendering process on a three-dimensional scene based on rendering parameters to obtain a rendered image, the rendering parameters including first rendering parameters obtained from a terminal device; selecting a first intermediate rendering result based on intermediate rendering results generated in the rendering process; generating a virtual reference frame based on the first intermediate rendering result; predicting the rendered image based on the virtual reference frame to obtain a predicted image; encoding a residual image between the predicted image and the rendered image, encoding encoded data of the residual image into a bitstream, encoding a second intermediate rendering result, and encoding encoded data of the second intermediate rendering result into a bitstream, where the second intermediate rendering result is a part of the first intermediate rendering result.

[0081] In this way, partial rendering is performed by the terminal device, and the server only needs to send some intermediate rendering results to the terminal device. Therefore, the present application can reduce interaction delay while ensuring that the bit rate overhead of the data stream sent by the server to the terminal device is effectively reduced. In addition, there is a strong correlation between the intermediate rendering results and the rendered image. Therefore, the present application can guarantee the image reconstruction quality because the rendered image is encoded based on the intermediate rendering results.

[0082] According to a fifth aspect, the bitstream further includes a first instruction identifier, the first instruction identifier indicating a type and subtype of the second intermediate rendering result. In this way, the terminal device knows the specific type and subtype of the third intermediate rendering result to be generated. The third intermediate rendering result is a part of the first intermediate rendering result other than the second intermediate rendering result.

[0083] For example, the second intermediate rendering result may also include multiple types, such as a motion vector type and an image type. If the second intermediate rendering result is CGMV, the corresponding type may be a motion vector type. If the second intermediate rendering result is an intermediate rendered image, the corresponding type may be an image type.

[0084] For example, each type of second intermediate rendering result may include multiple subtypes.

[0085] For example, the motion vector type may include subtype A1, subtype A2, and subtype A3, where the subtype corresponding to the CGMV of a static object is subtype A1, the subtype corresponding to the CGMV of a rigid dynamic object is subtype A2, and the subtype corresponding to the CGMV of a non-rigid dynamic object is subtype A3.

[0086] For example, an image type may include subtype B1 and subtype B2, where the subtype corresponding to areas corresponding to the four corners in the intermediate rendered image is subtype B1, and the subtype corresponding to areas other than the areas corresponding to the four corners in the intermediate rendered image is subtype B2.

[0087] According to a fifth aspect or any one of the implementations of the fifth aspect, the rendering parameters further include second rendering parameters generated by the server. The method further includes encoding third rendering parameters into the bitstream. The third rendering parameters include all or part of the parameters of the second rendering parameters.

[0088] According to the fifth aspect or any one of the implementations of the fifth aspect, the rendering parameters further include second rendering parameters generated by the server, and the bitstream further includes a second instruction identifier and / or a third instruction identifier, the second instruction identifier indicating whether the bitstream includes the third rendering parameters, the third rendering parameters including all or part of the parameters of the second rendering parameters, and the third instruction identifier indicating the type of the third rendering parameters.

[0089] According to the fifth aspect or any one of the implementations of the fifth aspect, the step of generating a virtual reference frame based on the first intermediate rendering result includes the steps of determining a type of the first intermediate rendering result and generating a virtual reference frame based on the first intermediate rendering result and the type of the first intermediate rendering result.

[0090] According to the fifth aspect or any one of the implementations of the fifth aspect, the step of generating a virtual reference frame based on the first intermediate rendering result and a type of the first intermediate rendering result includes, when the type of the first intermediate rendering result is a motion vector type, generating the virtual reference frame based on the first intermediate rendering result and the reconstructed image, wherein the first intermediate rendering result is a computer graphics motion vector CGMV, and the CGMV is used to describe a displacement relationship between samples in the rendered image and samples in the reconstructed image.

[0091] According to the fifth aspect or any one of the implementations of the fifth aspect, the step of generating a virtual reference frame based on the first intermediate rendering result and a type of the first intermediate rendering result includes a step of determining the intermediate rendering result as the virtual reference frame when the type of the first intermediate rendering result is determined to be an image type, the first intermediate rendering result is an intermediate rendered image, and a computational complexity corresponding to the intermediate rendered image is lower than a computational complexity corresponding to the rendered image.

[0092] According to the fifth aspect or any one of the implementations of the fifth aspect, the first intermediate rendering result includes a CGMV and / or an intermediate rendered image, and the computational complexity corresponding to the intermediate rendered image is lower than the computational complexity corresponding to the rendered image.

[0093] The fifth aspect and any one of the implementations of the fifth aspect correspond to the second aspect and any one of the implementations of the second aspect. For technical effects corresponding to the fifth aspect and any one of the implementations of the fifth aspect, please refer to the technical effects corresponding to the second aspect and any one of the implementations of the second aspect. Details will not be described again here.

[0094] According to a sixth aspect, the present application further provides a decoding method applicable to a terminal device, the decoding method including the steps of: receiving a bitstream; analyzing the bitstream to obtain an analysis result, the analysis result including a residual image corresponding to a current frame and a second intermediate rendering result corresponding to the current frame; performing a rendering process on a three-dimensional scene based on rendering parameters corresponding to the current frame to generate a third intermediate rendering result based on the second intermediate rendering result in the rendering process, the rendering parameters including the first rendering parameters generated by the terminal device; generating a virtual reference frame based on the first intermediate rendering result, the first intermediate rendering result including the second intermediate rendering result and the third intermediate rendering result; predicting the current frame based on the virtual reference frame to determine a predicted image; and performing reconstruction based on the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0095] In this way, the first intermediate rendering result obtained by the terminal device includes the third intermediate rendering result generated by the terminal device and the second intermediate rendering result generated by the server. Because the second intermediate rendering result generated by the server is more accurate than the second intermediate rendering result generated by the terminal device, the first intermediate rendering result used by the terminal device to generate the virtual reference frame is more accurate, which can further improve the accuracy of the virtual reference frame and the image quality of the reconstructed image. In addition, because the terminal device only needs to generate a portion of the first intermediate rendering result, the computing power of the terminal device can be saved and decoding efficiency can be improved.

[0096] It should be understood that if the server performs lossy encoding on the second intermediate result, the second intermediate result obtained by the terminal device through analysis is different from the second intermediate result encoded by the server, and if the server performs lossless encoding on the second intermediate result, the second intermediate result obtained by the terminal device through analysis is the same as the second intermediate result encoded by the server.

[0097] It should be understood that since the third intermediate rendering result is generated by the terminal device, the type of the intermediate result included in the first intermediate rendering result obtained by the terminal device through combination is the same as the type of the intermediate result included in the first intermediate rendering result generated by the server, and the accuracy of the intermediate result included in the first intermediate rendering result obtained by the terminal device through combination is equal to or less than the accuracy of the intermediate result included in the first intermediate rendering result generated by the server.

[0098] According to a sixth aspect, the decoded result further includes a first instruction identifier, the first instruction identifier indicating the type and subtype of the second intermediate rendering result, and the step of generating a third intermediate rendering result based on the second intermediate rendering result in the rendering processing step includes the steps of determining the type and subtype of the second intermediate rendering result based on the first instruction identifier, and generating a third intermediate rendering result of another subtype in the rendering processing step based on the type and subtype of the second intermediate rendering result, wherein the other subtype is one of the subtypes included in the type corresponding to the second intermediate rendering result other than the subtype of the second intermediate rendering result.

[0099] According to the sixth aspect or any one of the implementations of the sixth aspect, the analysis result further includes a second instruction identifier and a third instruction identifier, where the second instruction identifier indicates whether the bitstream includes a third rendering parameter, and the third identifier indicates a type of the third rendering parameter. If it is determined that the bitstream includes the third rendering parameter based on the second instruction identifier, the method further includes determining a type of the third rendering parameter based on the third instruction identifier and generating a fourth rendering parameter of another type based on the type of the third rendering parameter. The another type is one type other than the type of the third rendering parameter among a plurality of types corresponding to the second rendering parameter generated by the server. The rendering parameters further include the third rendering parameter and the fourth rendering parameter.

[0100] According to a sixth aspect or any one of the implementations of the sixth aspect, the method further includes, when it is determined based on the second instruction identifier that the bitstream does not include the third rendering parameter, generating, by the terminal device, second rendering parameters corresponding to the current frame, wherein the rendering parameters further include the second rendering parameters generated by the terminal device.

[0101] According to the sixth aspect or any one of the implementations of the sixth aspect, the step of generating a virtual reference frame based on the first intermediate rendering result includes the steps of determining a type of the first intermediate rendering result and generating a virtual reference frame based on the first intermediate rendering result and the type of the first intermediate rendering result.

[0102] According to the sixth aspect or any one of the implementations of the sixth aspect, the step of generating a virtual reference frame based on the first intermediate rendering result and the type information of the first intermediate rendering result includes, when the type of the first intermediate rendering result is a motion vector, generating the virtual reference frame based on the first intermediate rendering result and a reconstructed image of another frame, wherein the first intermediate rendering result is a computer graphics motion vector CGMV, and the CGMV is used to describe a displacement relationship between samples in the reconstructed image of the current frame and samples in the reconstructed image of another frame.

[0103] According to the sixth aspect or any one of the implementations of the sixth aspect, the step of generating a virtual reference frame based on the first intermediate rendering result and type information of the first intermediate rendering result includes a step of determining the first intermediate rendering result as the virtual reference frame when the type of the first intermediate rendering result is determined to be an image type, the first intermediate rendering result is an intermediate rendered image, and a computational complexity corresponding to the intermediate rendered image is lower than a computational complexity corresponding to the rendered image.

[0104] According to the sixth aspect or any one of the implementations of the sixth aspect, the first intermediate rendering result includes a CGMV and / or an intermediate rendered image, and the computational complexity corresponding to the intermediate rendered image is lower than the computational complexity corresponding to the rendered image.

[0105] The sixth aspect and any one of the implementations of the sixth aspect correspond to the third aspect and any one of the implementations of the third aspect. For technical effects corresponding to the sixth aspect and any one of the implementations of the sixth aspect, please refer to the technical effects corresponding to the third aspect and any one of the implementations of the third aspect. Details will not be described again here.

[0106] According to a seventh aspect, the present application provides a bitstream generation method, in which a bitstream is generated based on a first rendering parameter in any one of the third aspect and the implementations of the third aspect.

[0107] According to an eighth aspect, the present application provides a bitstream generation method, in which a bitstream is generated based on a first rendering parameter in any one of the sixth aspect and the implementations of the sixth aspect.

[0108] According to a ninth aspect, an embodiment of the present application provides a server including a memory and a processor, the memory coupled to the processor, the memory storing program instructions that, when executed by the processor, enable the server to perform an encoding method according to the second aspect or any one of possible implementations of the second aspect, or enable the server to perform an encoding method according to the fifth aspect or any one of possible implementations of the fifth aspect.

[0109] The ninth aspect and any one of the implementations of the ninth aspect correspond to any one of the second aspect and the implementations of the second aspect, or correspond to any one of the fifth aspect and the implementations of the fifth aspect. For technical effects corresponding to the ninth aspect and any one of the implementations of the ninth aspect, please refer to the technical effects corresponding to the second aspect and any one of the implementations of the second aspect, or refer to the technical effects corresponding to the fifth aspect and any one of the implementations of the fifth aspect. Details will not be described again here.

[0110] According to a tenth aspect, an embodiment of the present application provides a terminal device including a memory and a processor. The memory is coupled to the processor. The memory stores program instructions. When the program instructions are executed by the processor, the terminal device is capable of performing a decoding method according to the third aspect or any one of possible implementations of the third aspect, or the terminal device is capable of performing a decoding method according to the sixth aspect or any one of possible implementations of the sixth aspect.

[0111] The tenth aspect and any one of the implementations of the tenth aspect correspond to the third aspect and any one of the implementations of the third aspect, or correspond to the sixth aspect and any one of the implementations of the sixth aspect. For technical effects corresponding to the tenth aspect and any one of the implementations of the tenth aspect, please refer to the technical effects corresponding to the second aspect and any one of the implementations of the second aspect, or refer to the technical effects corresponding to the sixth aspect and any one of the implementations of the sixth aspect. Details will not be described again here.

[0112] According to an eleventh aspect, an embodiment of the present application provides a chip including one or more interface circuits and one or more processors. The interface circuits are configured to receive a signal from a storage device of a server and transmit the signal to the processor. The signal includes computer instructions stored in the storage device. Execution of the computer instructions by the processor enables the server to perform an encoding method according to the second aspect or any one of the possible implementations of the second aspect, or enables the server to perform an encoding method according to the fifth aspect or any one of the possible implementations of the fifth aspect.

[0113] The eleventh aspect and any one of the implementations of the eleventh aspect correspond to any one of the implementations of the second aspect and the second aspect, or correspond to any one of the implementations of the fifth aspect and the fifth aspect. For technical effects corresponding to the eleventh aspect and any one of the implementations of the eleventh aspect, please refer to the technical effects corresponding to the second aspect and any one of the implementations of the second aspect, or refer to the technical effects corresponding to the fifth aspect and any one of the implementations of the fifth aspect. Details will not be described again here.

[0114] According to a twelfth aspect, an embodiment of the present application provides a chip including one or more interface circuits and one or more processors. The interface circuits are configured to receive a signal from a storage device of a terminal device and transmit the signal to the processor. The signal includes computer instructions stored in the storage device. Execution of the computer instructions by the processor enables the terminal device to perform a decoding method according to the third aspect or any one of possible implementations of the third aspect, or enables the terminal device to perform a decoding method according to the sixth aspect or any one of possible implementations of the sixth aspect.

[0115] The twelfth aspect and any one of the implementations of the twelfth aspect correspond to the third aspect and any one of the implementations of the third aspect, or correspond to the sixth aspect and any one of the implementations of the sixth aspect. For technical effects corresponding to the twelfth aspect and any one of the implementations of the twelfth aspect, please refer to the technical effects corresponding to the third aspect and any one of the implementations of the third aspect, or refer to the technical effects corresponding to the sixth aspect and any one of the implementations of the sixth aspect. Details will not be described again here.

[0116] According to a thirteenth aspect, an embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium storing a computer program, which, when executed on a computer or a processor, enables the computer or the processor to perform an encoding method according to the second aspect or any one of possible implementations of the second aspect, or enables the computer or the processor to perform an encoding method according to the fifth aspect or any one of possible implementations of the fifth aspect.

[0117] The thirteenth aspect and any one of the implementations of the thirteenth aspect correspond to any one of the implementations of the second aspect and the second aspect, or correspond to any one of the implementations of the fifth aspect and the fifth aspect. For technical effects corresponding to the thirteenth aspect and any one of the implementations of the thirteenth aspect, please refer to the technical effects corresponding to the second aspect and any one of the implementations of the second aspect, or refer to the technical effects corresponding to the fifth aspect and any one of the implementations of the fifth aspect. Details will not be described again here.

[0118] According to a fourteenth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program that, when executed on a computer or a processor, enables the computer or processor to perform the decoding method according to the third aspect or any one of the possible implementations of the third aspect, or the computer or processor to perform the decoding method according to the sixth aspect or any one of the possible implementations of the sixth aspect.

[0119] The fourteenth aspect and any one of the implementations of the fourteenth aspect correspond to the third aspect and any one of the implementations of the third aspect, or correspond to the sixth aspect and any one of the implementations of the sixth aspect. For technical effects corresponding to the fourteenth aspect and any one of the implementations of the fourteenth aspect, please refer to the technical effects corresponding to the third aspect and any one of the implementations of the third aspect, or refer to the technical effects corresponding to the sixth aspect and any one of the implementations of the sixth aspect. Details will not be described again here.

[0120] According to a fifteenth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising computational instructions that, when executed by a computer or processor, enable the computer or processor to perform an encoding method according to the second aspect or any one of possible implementations of the second aspect, or enable the computer or processor to perform an encoding method according to the fifth aspect or any one of possible implementations of the fifth aspect.

[0121] The fifteenth aspect and any one of the implementations of the fifteenth aspect correspond to any one of the second aspect and the implementations of the second aspect, or correspond to any one of the fifth aspect and the implementations of the fifth aspect. For technical effects corresponding to the fifteenth aspect and any one of the implementations of the fifteenth aspect, please refer to the technical effects corresponding to the second aspect and any one of the implementations of the second aspect, or refer to the technical effects corresponding to the fifth aspect and any one of the implementations of the fifth aspect. Details will not be described again here.

[0122] According to a sixteenth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising computational instructions that, when executed by a computer or processor, enable the computer or processor to perform a decoding method according to the third aspect or any one of possible implementations of the third aspect, or enable the computer or processor to perform a decoding method according to the sixth aspect or any one of possible implementations of the sixth aspect.

[0123] The sixteenth aspect and any one of the implementations of the sixteenth aspect correspond to the third aspect and any one of the implementations of the third aspect, or correspond to the sixth aspect and any one of the implementations of the sixth aspect. For technical effects corresponding to the sixteenth aspect and any one of the implementations of the sixteenth aspect, please refer to the technical effects corresponding to the third aspect and any one of the implementations of the third aspect, or refer to the technical effects corresponding to the sixth aspect and any one of the implementations of the sixth aspect. Details will not be described again here.

[0124] According to a seventeenth aspect, an embodiment of the present application provides an apparatus for storing a bitstream. The apparatus includes a receiver and at least one storage medium. The receiver is configured to receive the bitstream, and the at least one storage medium is configured to store the bitstream. The bitstream is generated according to any one of the second aspect and implementations of the second aspect, or the fifth aspect and implementations of the fifth aspect, or the seventh aspect, or the eighth aspect.

[0125] The seventeenth aspect and any one of the implementations of the seventeenth aspect correspond to any one of the second aspect and any one of the implementations of the second aspect, or correspond to any one of the fifth aspect and any one of the implementations of the fifth aspect. For technical effects corresponding to the seventeenth aspect and any one of the implementations of the seventeenth aspect, please refer to the technical effects corresponding to the second aspect and any one of the implementations of the second aspect, or refer to the technical effects corresponding to the fifth aspect and any one of the implementations of the fifth aspect. Details will not be described again here.

[0126] According to an eighteenth aspect, an embodiment of the present application provides an apparatus for transmitting a bitstream. The apparatus includes a transmitter and at least one storage medium, where the at least one storage medium is configured to store a bitstream, the bitstream being generated according to any one of the second aspect and implementations of the second aspect, or the fifth aspect and implementations of the fifth aspect, or the seventh aspect, or the eighth aspect. The transmitter is configured to obtain the bitstream from the storage medium and transmit the bitstream to a device-side device through a transmission medium.

[0127] The 18th aspect and any one of the implementations of the 18th aspect correspond to the 2nd aspect and any one of the implementations of the 2nd aspect, or correspond to the 5th aspect and any one of the implementations of the 5th aspect. For technical effects corresponding to the 18th aspect and any one of the implementations of the 18th aspect, please refer to the technical effects corresponding to the 2nd aspect and any one of the implementations of the 2nd aspect, or refer to the technical effects corresponding to the 5th aspect and any one of the implementations of the 5th aspect. Details will not be described again here.

[0128] According to a nineteenth aspect, an embodiment of the present application provides a system for distributing bitstreams, the system including: at least one storage medium configured to store at least one bitstream, the at least one bitstream being generated according to any one of the second aspect and an implementation thereof, or the fifth aspect and any one of the implementation thereof, or the seventh aspect, or the eighth aspect; and a streaming media device configured to obtain the target bitstream from the at least one storage medium and transmit the target bitstream to a device-side device, the streaming media device including a content server or a content distribution server.

[0129] The 19th aspect and any one of the implementations of the 19th aspect correspond to the 2nd aspect and any one of the implementations of the 2nd aspect, or correspond to the 5th aspect and any one of the implementations of the 5th aspect. For technical effects corresponding to the 19th aspect and any one of the implementations of the 19th aspect, please refer to the technical effects corresponding to the 2nd aspect and any one of the implementations of the 2nd aspect, or refer to the technical effects corresponding to the 5th aspect and any one of the implementations of the 5th aspect. Details will not be described again here. [Brief explanation of the drawings]

[0130] [Figure 1a] FIG. 1 is a diagram of an exemplary application scenario. [Figure 1b] FIG. 1 is a diagram of an exemplary device-cloud collaboration system. [Figure 2a] FIG. 2 is a diagram of an exemplary encoding process. [Figure 2b] FIG. 2 is a diagram of an exemplary encoding mechanism. [Figure 3a] FIG. 1 is a diagram of an exemplary decoding process. [Figure 3b] FIG. 1 is a diagram of an exemplary decoding mechanism. [Figure 4] FIG. 2 is a diagram of an exemplary encoding process. [Figure 5] FIG. 1 is a diagram of an exemplary decoding process. [Figure 6a] FIG. 2 is a diagram of an exemplary encoding process. [Figure 6b] FIG. 1 is a diagram of an example model-view projection transformation. [Figure 7] FIG. 1 is a diagram of an exemplary decoding process. [Figure 8] FIG. 2 is a diagram of an exemplary encoding process. [Figure 9] FIG. 1 is a diagram of an exemplary decoding process. [Figure 10] FIG. 2 is a diagram of an exemplary encoding process. [Figure 11] FIG. 1 is a diagram of an exemplary decoding process. [Figure 12] FIG. 2 is a diagram of an exemplary encoding process. [Figure 13] FIG. 1 is a diagram of an exemplary decoding process. [Figure 14] FIG. 1 is a diagram of an exemplary device structure. DETAILED DESCRIPTION OF THE INVENTION

[0131] The following clearly and completely describes the technical solutions of the embodiments of the present application with reference to the accompanying drawings of the embodiments of the present application. It is clear that the described embodiments are only a part, but not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0132] The term "and / or" in this specification only describes a relational relationship for describing related objects and represents a three-way relationship. For example, A and / or B can represent the following three cases: when only A is present, when both A and B are present, and when only B is present.

[0133] In the description and claims of the embodiments of the present application, terms such as "first" and "second" are intended to distinguish between different objects, but do not indicate a particular order of the objects. For example, "first object," "second object," etc. are used to distinguish between different objects, but do not indicate a particular order of the objects.

[0134] In the embodiments of the present application, words such as "example" or "for example" represent providing an example, illustration, or explanation. Any embodiment or design solution described in the embodiments of the present application as an "example" or "for example" should not be described as being preferred over another embodiment or design solution, or as having more advantages than another embodiment or design solution. Specifically, words such as "example" or "for example" are used to present related concepts in a particular way.

[0135] In describing the embodiments of this application, unless otherwise specified, "plurality" means two or more. For example, a plurality of processing units is two or more processing units, and a plurality of systems is two or more systems.

[0136] 1a is a diagram of an exemplary application scenario. The application scenario shown in FIG. 1a is a device-cloud collaborative rendering scenario. The present application can be applied to various device-cloud collaborative scenarios, such as cloud gaming and VR / AR conferencing, which are not limited in the present application. In the present application, cloud gaming is used as an example for explanation.

[0137] As shown in FIG. 1a, in one possible manner, a terminal device, such as a mobile phone, a personal computer (PC), VR glasses, an AR helmet, a tablet computer, or another terminal device, can send first rendering parameters to a central server (e.g., a game server) through an edge server. The central server then performs a rendering process on the three-dimensional scene based on the rendering parameters (which may include only the first rendering parameters uploaded by the terminal device, or may include the first rendering parameters uploaded by the terminal device and second rendering parameters generated by the central server) to obtain a rendered image (e.g., a game image). The central server then encodes the rendered image based on a portion of the intermediate rendering result (hereinafter referred to as the first intermediate rendering result) generated in the rendering process to obtain a bitstream, and distributes the bitstream to each terminal device through the edge server. Then, the terminal device can perform a rendering process on the three-dimensional scene based on the rendering parameters (the rendering parameters include a first rendering parameter and a fifth rendering parameter, where the fifth rendering parameter is also generated by the terminal device, and the fifth rendering parameter may include the same parameters as the second rendering parameter), and generate a first intermediate rendering result in the rendering process. Then, the terminal device decodes the bitstream based on the first intermediate rendering result to obtain a reconstructed image.

[0138] As shown in FIG. 1a, in one possible manner, a terminal device, such as a mobile phone, a PC, VR glasses, an AR helmet, a tablet computer, or another terminal device, can send first rendering parameters to a central server through an edge server. The central server then performs a rendering process on the three-dimensional scene based on the rendering parameters (which may include only the first rendering parameters uploaded by the terminal device, or may include the first rendering parameters uploaded by the terminal device and second rendering parameters generated by the server) to obtain a rendered image. The central server then encodes the rendered image based on the first intermediate rendering result generated in the rendering process, encodes the second intermediate rendering result (which is a part of the first intermediate rendering result) to obtain a bitstream, and distributes the bitstream to each terminal device through the edge server. Thereafter, the terminal device may perform a rendering process on the three-dimensional scene based on the rendering parameters (the rendering parameters include the first rendering parameter and the fifth rendering parameter), and generate a third intermediate rendering result (the third intermediate rendering result is a part of the first intermediate rendering result other than the second intermediate rendering result) in the rendering process. Then, the terminal device may generate a first intermediate rendering result based on the second intermediate rendering result and the third intermediate rendering result, and decode the bitstream based on the first intermediate rendering result to obtain a reconstructed image.

[0139] As shown in FIG. 1a, in one possible manner, a terminal device, such as a mobile phone, a PC, VR glasses, an AR helmet, a tablet computer, or another terminal device, can send first rendering parameters to a central server through an edge server. The central server then performs a rendering process on the three-dimensional scene based on the rendering parameters (the rendering parameters include the first rendering parameters uploaded by the terminal device and the second rendering parameters generated by the central server) to obtain a rendered image. The central server then encodes the rendered image based on the first intermediate rendering result generated in the rendering process to obtain a bitstream, and encodes third rendering parameters (the third rendering parameters include all or part of the parameters of the second rendering parameters) into the bitstream. The central server then distributes the bitstream to each terminal device through the edge server. Then, the terminal device can perform a rendering process on the three-dimensional scene based on the rendering parameters (if the third rendering parameters include all of the parameters of the second rendering parameters, the rendering parameters may include the first rendering parameters and the third rendering parameters obtained from the bitstream by analysis, or if the third rendering parameters include some of the parameters of the second rendering parameters, the rendering parameters may include the first rendering parameters, the third rendering parameters obtained from the bitstream by analysis, and a fourth rendering parameter generated by the terminal device, where the fourth rendering parameter is a parameter of the second rendering parameters other than the third rendering parameter), and generate a first intermediate rendering result in the rendering process. Then, the terminal device decodes the bitstream based on the first intermediate rendering result to obtain a reconstructed image.

[0140] As shown in FIG. 1a, in one possible manner, a terminal device, such as a mobile phone, a PC, VR glasses, an AR helmet, a tablet computer, or another terminal device, can send first rendering parameters to a central server through an edge server. The central server then performs a rendering process on the three-dimensional scene based on the rendering parameters (including the first rendering parameters uploaded by the terminal device and the second rendering parameters generated by the central server) to obtain a rendered image. The central server then encodes the rendered image based on the first intermediate rendering result generated in the rendering process, encodes the second intermediate rendering result to obtain a bitstream, and encodes the third rendering parameters into the bitstream. The central server then distributes the bitstream to each terminal device through the edge server. Then, the terminal device can perform a rendering process on the three-dimensional scene based on the rendering parameters (if the third rendering parameters include all of the parameters of the second rendering parameters, the rendering parameters may include the first rendering parameters and the third rendering parameters obtained from the bitstream by analysis, or if the third rendering parameters include some of the parameters of the second rendering parameters, the rendering parameters may include the first rendering parameters, the third rendering parameters obtained from the bitstream by analysis, and a fourth rendering parameter generated by the terminal device), and generate a third intermediate rendering result in the rendering process. Then, the terminal device generates a first intermediate rendering result based on the second intermediate rendering result and the third intermediate rendering result, and decodes the bitstream based on the first intermediate rendering result to obtain a reconstructed image.

[0141] In this way, some or all of the rendering is performed by the terminal device, and the server does not need to transmit intermediate rendering results or only some of the intermediate rendering results to the terminal device. Therefore, in the present application, interaction delay can be reduced while ensuring that the bitrate overhead of the data stream transmitted by the server to the terminal device is effectively reduced. In addition, there is a strong correlation between the intermediate rendering results and the rendered image. Therefore, in the present application, the rendered image is encoded based on the intermediate rendering results, so that image reconstruction quality can be guaranteed. In addition, the data amount of the rendering parameters is much smaller than the data amount of the intermediate rendering results. Therefore, even if the server in the present application further transmits rendering parameters to the terminal device, the bitrate overhead of the data stream transmitted by the server to the terminal device in the present application is smaller than the bitrate overhead of the data stream transmitted in the prior art.

[0142] For example, the rendering parameters may be all parameters input to the graphics rendering engine and required for rendering processing by the graphics rendering engine, and may include various parameters used for rendering, position vectors and color vectors of all light sources, position vectors of the player or observer, information such as the sampling method of each texture and the position coordinates of objects in each scene, motion tracks of moving objects, skeletal animation parameters, etc. This is not limited to this application. The first rendering parameters and the second rendering parameters can form the rendering parameters (i.e., all parameters input to the graphics rendering engine and required for rendering by the graphics rendering engine). It should be noted that the type of parameters included in the fifth rendering parameters generated by the terminal device is the same as the type of parameters included in the second rendering parameters generated by the central server, and the precision of the parameters included in the fifth rendering parameters is equal to or less than the precision of the parameters included in the second rendering parameters.

[0143] For example, the intermediate rendering result may be intermediate data used to generate the image / video to be displayed and generated by a graphics rendering engine in the process of generating the image (i.e., the rendered image) / video (i.e., the rendered video) to be displayed. For example, the intermediate rendering results may include, but are not limited to, computer graphics motion vectors (CGMV), intermediate rendered images (an intermediate rendered image is an image generated before a final rendered image (i.e., the aforementioned rendered image) is generated, and the computational complexity of the intermediate rendered image is lower than that of a rendered image, and the intermediate rendered image may be, for example, an intermediate rendered image on which indirect lighting rendering has not been performed, an intermediate rendered image on which specular reflection processing has not been performed, or an intermediate rendered image on which highlight processing has not been performed), a position map, a normal map, an albedo map, a specular intensity map, a mesh identifier (Mesh ID), a material ID (each material map corresponds to one material ID), a render ID (each object (or one three-dimensional object model) corresponds to one Render ID), depth information, etc. This is not limited in the present application. The first intermediate rendering result is a part of all intermediate rendering results generated in the rendering process, and the second intermediate rendering result and the third intermediate rendering result can form the first intermediate rendering result. It should be noted that the type of the intermediate result included in the first intermediate rendering result generated by the terminal device is the same as the type of the intermediate result included in the first intermediate rendering result generated by the central server, and the accuracy of the intermediate result included in the first intermediate rendering result generated by the terminal device is equal to or less than the accuracy of the intermediate result included in the first intermediate rendering result generated by the central server.

[0144] Based on the application scenario of FIG. 1a, the present application provides a device-cloud collaboration system, which can be shown in FIG. 1b.

[0145] 1b is a diagram of an exemplary device-cloud collaboration system. The device-cloud collaboration system shown in FIG. 1b may include a server and a terminal device.

[0146] For example, the server may be the central server of Figure 1a, or the server may be a single server or a server cluster, as shown in Figure 1b, which is not limited in this application.

[0147] For example, the server may include a first rendering module, an encoder, and a first communication module. It should be understood that Fig. 1b is only an example of the present application. The server in the present application may include more or fewer modules than those shown in Fig. 1b. This is not a limitation of the present application.

[0148] For example, the first rendering module may be configured to perform a rendering process on the three-dimensional scene based on the rendering parameters to obtain a rendered image and an intermediate rendering result.

[0149] For example, the encoder may be configured to encode a rendered image based on a first intermediate rendering result, encode a second intermediate rendering result, and encode second rendering parameters into the bitstream.

[0150] For example, the first communication module may be configured to communicate with another electronic device, for example to transmit the bitstream output by the encoder to a terminal device.

[0151] For example, terminal devices include, but are not limited to, personal computers, computer workstations, smartphones, tablet computers, servers, smart cameras, intelligent vehicles, other types of mobile phones, media consumption devices, wearable devices (e.g., VR / AR helmets or VR glasses), set-top boxes, game consoles, etc.

[0152] Further, as shown in Fig. 1b, for example, the terminal device may include a second communication module, a second rendering module, a decoder, and a display module. It should be understood that Fig. 1b is only an example of the present application. The terminal device in the present application may include more or fewer modules than those shown in Fig. 1b. This is not limited in the present application.

[0153] For example, the second communication module may be configured to communicate with another electronic device, for example to receive a bitstream transmitted by a server.

[0154] For example, the second rendering module may be configured to perform a rendering process on the three-dimensional scene based on the rendering parameters to obtain a first intermediate rendering result or a third intermediate rendering result. It should be understood that the second rendering module may also generate a rendered image, but the rendered image is not used for display by a subsequent display module.

[0155] For example, the decoder may be configured to decode the bitstream based on the first intermediate rendering result to obtain a reconstructed image.

[0156] For example, the display module may be configured to display the reconstructed image.

[0157] It should be understood that the video encoding standard used by the encoder and decoder is not limited in this application. For example, the video encoding standard may include, but is not limited to, H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), H.266 / VVC (Versatile Video Coding), AV1 (AOMedia Video 1, "AOMedia" is a video encoding standard developed by the Alliance for Open Media), etc., and extension standards of these video encoding standards. In addition, the video encoding standard may further include new video encoding standards and extension standards created with the development of video encoding and decoding technologies.

[0158] In the following, the encoding and decoding steps based on Figs. 1a and 1b are described.

[0159] 2a is a diagram of an exemplary encoding process. In the embodiment of FIG. 2a, the server does not send the first intermediate rendering result to the terminal device, nor does it send the second rendering parameters to the terminal device. In the embodiment of FIG. 2a, an example in which one frame of rendered image is encoded is used for explanation.

[0160] S201: A server performs a rendering process on a three-dimensional scene based on rendering parameters to obtain a rendered image, where the rendering parameters include a first rendering parameter obtained from a terminal device.

[0161] For example, the server may obtain rendering parameters and then input the rendering parameters to a graphics rendering engine. The graphics rendering engine performs a rendering process on the three-dimensional scene based on the rendering parameters to obtain a rendered image. The graphics rendering engine belongs to the first rendering module in FIG. 1b.

[0162] For example, the server can obtain first rendering parameters (the first rendering parameters are generated by the terminal device) from the terminal device. In one possible manner, if the first rendering parameters generated by the terminal device are all parameters required for the rendering process by the graphics rendering engine, the server can determine the first rendering parameters as the rendering parameters. That is, the rendering parameters obtained by the server can include only the first rendering parameters. In one possible manner, if the first rendering parameters generated by the terminal device are only a part of the parameters required for the rendering process by the graphics rendering engine, the server can further generate second rendering parameters and then determine the first rendering parameters and the second rendering parameters as the rendering parameters. That is, the rendering parameters obtained by the server can include the first rendering parameters and the second rendering parameters. The first rendering parameters and the second rendering parameters will be specifically described in the following embodiments.

[0163] It should be noted that the server can obtain the first rendering parameters from the terminal device based on a predetermined cycle. The predetermined cycle may be set based on requirements, which is not limited in the present application. It should be understood that the first rendering parameters obtained by the server from the terminal device each time may be used to render images of one or more frames.

[0164] For example, the rendering process of a graphics rendering engine may be as follows: an image engine may perform geometric transformation, projection transformation, perspective transformation, and window clipping on a three-dimensional object model in a three-dimensional scene based on input rendering parameters, and then generate a rendered image based on the obtained material and light and shadow information.

[0165] S202: The server selects a first intermediate rendering result based on the intermediate rendering results generated in the rendering process.

[0166] For example, a graphics rendering engine in a first rendering module may generate intermediate rendering results in a rendering process (the rendering process is the period between when the graphics rendering engine obtains the rendering parameters and when the rendered image is obtained), and select a first intermediate rendering result from the intermediate rendering results, after which the rendered image may be encoded based on the first intermediate rendering result.

[0167] For example, some intermediate rendering results may be selected from the intermediate rendering results as the first intermediate rendering results. For example, in this application, a method for encoding a rendered image based on the first intermediate rendering result may be to generate a virtual reference frame based on the first intermediate rendering result, and then encode the rendered image based on the virtual reference frame. Furthermore, some intermediate rendering results that can be used to generate the virtual reference frame may be selected from the intermediate rendering results as the first intermediate rendering results. For example, CGMV, intermediate rendered images, etc. may be selected as the first intermediate rendering results. This is not a limitation in this application.

[0168] S203: The server generates a virtual reference frame based on the first intermediate rendering result.

[0169] For example, a virtual reference frame is a reference frame generated based on prior information, which includes information other than the decoded information in the encoder and the video / image to be encoded. To further remove video temporal redundancy, a virtual reference frame may be used as supplementary information for inter-coding.

[0170] For example, different first intermediate rendering results correspond to different types. For example, the type of the CGMV is a motion vector type, and the type of the intermediate rendered image is an image type. In one possible manner, the type of the first intermediate rendering result may be determined, and then a virtual reference frame is generated based on the first intermediate rendering result and the type of the first intermediate rendering result. Specifically, manners for generating a virtual reference frame for different types of first intermediate rendering results are described below.

[0171] It should be noted that S201 and S202 may be performed by the first rendering module of Fig. 1b, and S203 may be performed by the first rendering module of Fig. 1b, or may be performed by a module in the server other than the module shown in Fig. 1b. This is not limited in the present application (this application uses an example in which the first rendering module performs S203). S204 and S205 may be performed by the encoder of Fig. 1b.

[0172] S204: The server predicts the rendered image based on the virtual reference frame to obtain a predicted image.

[0173] S205: The server encodes a residual image between the predicted image and the rendered image, and encodes the encoded data of the residual image into a bitstream, where the bitstream does not include the encoded data of the first intermediate rendering result.

[0174] Figure 2b is a diagram of an exemplary encoding scheme. The embodiment of Figure 2b shows the encoding scheme of the encoder of Figure 1b. The video to be encoded in Figure 2b is a rendered video, which may include multiple frames of rendered images.

[0175] As shown in FIG. 2b, for example, the encoder may include a partitioning module, an intra-prediction module, an inter-prediction module, a transform module, a quantization module, an entropy coding module, an inverse quantization module, an inverse transform module, a loop filtering module, and a storage device.

[0176] For example, after generating the virtual reference frame, the first rendering module may insert the virtual reference frame into a reference frame list of a storage device and increase the length of the reference frame list by one.

[0177] 2b, the encoding process of the encoder may be as follows: after obtaining the to-be-encoded video, a division module of the encoder may divide the rendered image of each frame in the to-be-encoded video to obtain to-be-encoded blocks, and for each to-be-encoded block, a decision module (not shown in FIG. 2b) may decide whether to perform inter prediction or intra prediction on the to-be-encoded block.

[0178] If intra prediction is determined, the intra prediction module can perform intra prediction to determine a prediction block corresponding to the block to be coded. After the prediction block is obtained, a residual block between the block to be coded and the prediction block can be determined. The residual block is then input to a transformation module, which transforms the residual block to obtain a transformation result and outputs the transformation result to a quantization module. The quantization module can then quantize the transformation result to obtain a quantization result and output the quantization result to an entropy coding module. The entropy coding module can then perform entropy coding on the quantization result to obtain coded data of the residual block, and code the coded data of the residual block into a bitstream.

[0179] For example, the quantization module may further output the quantization result to an inverse quantization module, which performs inverse quantization to obtain an inverse quantization result and outputs the inverse quantization result to an inverse transform module. Next, the inverse transform module performs inverse transform on the inverse quantization result to obtain a decoded residual block. The decoded residual block may then be overlapped with the prediction block output by the intra prediction module to obtain a reconstructed block. Thereafter, the reconstructed block is input to a loop filtering module, which performs loop filtering on the reconstructed block to obtain a reconstructed block obtained by filtering, and outputs the reconstructed block obtained by filtering to a storage device. (After obtaining the reconstructed block obtained by filtering, the loop filtering module may splice the reconstructed block obtained by filtering at corresponding positions of the image to be reconstructed. After obtaining a reconstructed image of one frame, the loop filtering module may output the reconstructed image to a storage device for storage.)

[0180] If inter prediction is determined, an inter prediction module may perform intra prediction to determine a prediction block corresponding to the current block to be coded.

[0181] In one possible manner, the inter-prediction module can select a virtual reference frame from a sequence of reference frames in a storage device as a reference frame for the rendered image, and for a block to be coded in the rendered image, a matching prediction block (i.e., the aforementioned predicted image) can be searched for in the virtual reference frame.

[0182] In one possible manner, the inter prediction module can use both a virtual reference frame and an original reference frame in the reference frame sequence in the storage device as candidate reference frames for the rendered image. The original reference frame is a reconstructed image. For a block to be coded in the rendered image, inter prediction can be performed based on multiple candidate reference frames to determine multiple prediction blocks. One prediction block is determined correspondingly for each candidate reference frame. An optimal prediction block can be selected from the multiple prediction blocks (e.g., using rate-distortion cost as an evaluation criterion, the prediction block with the smallest rate-distortion cost can be determined as the optimal prediction block (i.e., the aforementioned predicted image)).

[0183] After the prediction block is obtained, a residual block (i.e., the aforementioned residual image) between the block to be coded and the prediction block can be determined. Then, the residual block is input to a transformation module, which transforms the residual block to obtain a transformation result and outputs the transformation result to a quantization module. Then, the quantization module can quantize the transformation result to obtain a quantization result and output the quantization result to an entropy coding module. Then, the entropy coding module can perform entropy coding on the quantization result to obtain coded data of the residual block, and code the coded data of the residual block into a bitstream (i.e., the bitstream obtained in S205).

[0184] For example, the quantization module may further output the quantization result to an inverse quantization module, which performs inverse quantization to obtain an inverse quantization result and outputs the inverse quantization result to an inverse transform module. Next, the inverse transform module performs inverse transform on the inverse quantization result to obtain a decoded residual block. The decoded residual block may then be overlapped with the prediction block output by the intra prediction module to obtain a reconstructed block. Thereafter, the reconstructed block is input to a loop filtering module, which performs loop filtering on the reconstructed block to obtain a reconstructed block obtained by filtering, and outputs the reconstructed block obtained by filtering to a storage device. (After obtaining the reconstructed block obtained by filtering, the loop filtering module may splice the reconstructed block obtained by filtering at corresponding positions of the image to be reconstructed. After obtaining a reconstructed image of one frame, the loop filtering module may output the reconstructed image to a storage device for storage.)

[0185] For example, to avoid an overly long reference frame list and reduce encoding efficiency, after the rendered image is encoded, a virtual reference frame in the reference frame list in the storage device may be deleted, and the length of the reference frame list is reduced by 1.

[0186] Fig. 3a is a diagram of an exemplary decoding process. Fig. 3a shows the decoding process corresponding to Fig. 2a. In the embodiment of Fig. 3a, a terminal device performs a rendering process based on locally generated rendering parameters to generate a first intermediate rendering result. In the embodiment of Fig. 3a, an example in which one frame of image is decoded is used for explanation.

[0187] S301: A terminal device receives a bitstream.

[0188] For example, the terminal device may receive the bitstream after the server transmits it.

[0189] S302: The terminal device parses the bitstream to obtain a parsing result, where the parsing result includes a residual image corresponding to the current frame.

[0190] Figure 3b is a diagram of an exemplary decoding mechanism.The embodiment of Figure 3b shows the decoding mechanism of the decoder of Figure 1b.

[0191] As shown in FIG. 3b, for example, the decoder may include an entropy decoding module, an inverse quantization module, an inverse transform module, an intra prediction module, an inter prediction module, a loop filtering module, and a storage device.

[0192] For example, after receiving a bitstream, the terminal device can analyze the bitstream to obtain an analysis result. The analysis process may be as follows: the coded data of the residual block is extracted from the bitstream, and then the coded data of the residual block is input to an entropy decoding module. The entropy decoding module performs entropy decoding on the coded data of the residual block to obtain entropy decoded data. Next, the entropy decoded data may be input to an inverse quantization module, which performs inverse quantization to obtain inverse quantized data, and inputs the inverse quantized data to an inverse transform module. Then, the inverse transform module performs inverse transform on the inverse quantized data to obtain a decoded residual block (i.e., a residual image corresponding to the current frame in S302).

[0193] It should be understood that if the encoder side performs lossy coding on the residual block, the residual block obtained by the decoder side through analysis will be different from the residual block coded by the encoder side, whereas if the encoder side performs lossless coding on the residual block, the residual block obtained by the decoder side through analysis will be the same as the residual block coded by the encoder side.

[0194] It should be noted that the bitstream description information used to describe the bitstream can be further extracted from the bitstream, in which case operations such as entropy decoding, inverse quantization, and inverse transform do not need to be performed on the bitstream description information, and the bitstream description information can be used in the subsequent decoding process.

[0195] S303: The terminal device performs a rendering process on the three-dimensional scene based on rendering parameters corresponding to the current frame, and generates a first intermediate rendering result in the rendering process, where the rendering parameters include the first rendering parameters generated by the terminal device.

[0196] For example, after a residual image corresponding to the current frame is obtained by analysis, a second rendering module (which may be a graphics rendering engine included in the second rendering module) can perform rendering processing on the three-dimensional scene based on rendering parameters corresponding to the current frame.

[0197] For example, if the first rendering parameters uploaded to the server by the terminal device are all parameters required for the rendering process by the graphics rendering engine, the second rendering module can perform the rendering process on the three-dimensional scene based on the first rendering parameters corresponding to the current frame. If the first rendering parameters uploaded by the terminal device are only a part of the parameters required for the rendering process by the graphics rendering engine, the terminal device can further generate fifth rendering parameters, and then the second rendering module can perform the rendering process on the three-dimensional scene based on the first rendering parameters corresponding to the current frame and the fifth rendering parameters.

[0198] Please note that the type of parameters included in the fifth rendering parameters generated by the terminal device is the same as the type of parameters included in the second rendering parameters generated by the central server, and the precision of the parameters included in the fifth rendering parameters is less than or equal to the precision of the parameters included in the second rendering parameters.

[0199] It should be noted that the type of intermediate result included in the first intermediate rendering result generated by the terminal device is the same as the type of intermediate result included in the first intermediate rendering result generated by the server, and the accuracy of the intermediate result included in the first intermediate rendering result generated by the terminal device is less than or equal to the accuracy of the intermediate result included in the first intermediate rendering result generated by the server.

[0200] It should be noted that a rendered image can be obtained even when a graphics rendering engine of a terminal device performs the rendering process. Because the computing power of the terminal device is lower than that of a server, the image quality (image quality may include subjective quality and objective quality (e.g., rate-distortion performance or PSNR (Peak Signal to Noise Ratio))) of a rendered image obtained by performing the rendering process by the graphics rendering engine of the terminal device is lower than the image quality of a rendered image obtained by performing the rendering process by the graphics rendering engine of the server. Therefore, the terminal device can display a reconstructed image obtained by decoding the bitstream, rather than displaying a rendered image obtained by performing the rendering process by the graphics rendering engine of the terminal device.

[0201] S304: The terminal device generates a virtual reference frame based on the first intermediate rendering result.

[0202] For example, regarding S304, please refer to the description of S203, and the details will not be described again here.

[0203] For example, after generating the virtual reference frame, the terminal device can reconstruct a residual image corresponding to the current frame based on the virtual reference frame to obtain a reconstructed image, which has higher image quality than the rendered image obtained by performing a rendering process using a graphics rendering engine of the terminal device, and then the terminal device can display the reconstructed image.

[0204] It should be noted that S301 to S303 may be performed by a second rendering module in the terminal device, and S304 may be performed by a second rendering module in the terminal device, or may be performed by a module in the terminal device other than the module shown in Fig. 1b (this application uses an example in which the second rendering module performs S304 for explanation). S305 and S306 may be performed by the decoder in Fig. 1b.

[0205] S305: The terminal device predicts the current frame based on the virtual reference frame to obtain a predicted image.

[0206] S306: The terminal device performs reconstruction based on the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0207] Again, as shown in FIG. 3b, for example, after generating the virtual reference frame, the second rendering module may insert the virtual reference frame into the reference frame list of the storage device and increase the length of the reference frame list by 1.

[0208] Again, as shown in FIG. 3b, for example, once intra prediction is determined based on bitstream description information obtained from the bitstream by analysis, an intra prediction module can perform intra prediction to obtain a predicted block. The predicted block can then be overlapped with the decoded residual block to obtain a reconstructed block. Loop filtering is then performed on the reconstructed block to obtain a filtered reconstructed block, which can be output to a storage device. (After obtaining the filtered reconstructed block, the loop filtering module can splice the filtered reconstructed block at a corresponding position of the image to be reconstructed; after obtaining a reconstructed image of one frame, the loop filtering module can output the reconstructed image to a storage device for storage.)

[0209] Again, as shown in FIG. 3b, for example, when inter-prediction is determined based on bitstream description information obtained from the bitstream by analysis, an inter-prediction module can perform inter-prediction to obtain a predicted block (i.e., a predicted image of S305).

[0210] In one possible manner, if the inter prediction module determines that a reference frame is a virtual reference frame based on bitstream description information obtained from the bitstream by analysis, the inter prediction module can select a corresponding virtual reference frame from the reference frame sequence in the storage device as the reference frame, and then determine a prediction block based on the virtual reference frame.

[0211] In one possible manner, if the inter prediction module determines that a reference frame is an original reference frame based on bitstream description information obtained from the bitstream by analysis, the inter prediction module can select a corresponding original reference frame from the reference frame sequence in the storage device as the reference frame, and then determine a prediction block based on the original reference frame.

[0212] After the prediction block is obtained, the prediction block may be overlapped with the residual block obtained by decoding to obtain a reconstructed block. Then, loop filtering is performed on the reconstructed block to obtain a reconstructed block obtained by filtering, and the reconstructed block obtained by filtering may be output to a storage device. (After obtaining the reconstructed block obtained by filtering, the loop filtering module may splice the reconstructed block obtained by filtering at a corresponding position of the image to be reconstructed, and after obtaining a reconstructed image of one frame, the loop filtering module may output the reconstructed image to a storage device.)

[0213] It should be understood that the terminal device may further generate a bitstream based on the first rendering parameters and then transmit the bitstream to the server based on a predetermined period. In this way, the server can obtain the first rendering parameters. In addition, since the data amount of the first rendering parameters is small, the terminal device can encode the first rendering parameters, encode the encoded data of the first rendering parameters into a bitstream, and then transmit the bitstream to the server. Alternatively, the terminal device can not encode the first rendering parameters, but directly encode the first rendering parameters into a bitstream and transmit the bitstream to the server. This is not limited in the present application.

[0214] In this way, in the present application, all rendering is performed by the terminal device, and the server does not need to send intermediate rendering results to the terminal device. Therefore, in the present application, interaction delay can be reduced while ensuring that the bit rate overhead of the data stream sent by the server to the terminal device is effectively reduced. In addition, there is a strong correlation between the intermediate rendering results and the rendered image. Therefore, in the present application, the rendered image is encoded based on the intermediate rendering results, so that the image reconstruction quality can be guaranteed.

[0215] 4 is a diagram of an exemplary encoding process. In the embodiment of FIG. 4, the server sends a part of the first intermediate rendering result (hereinafter referred to as the second intermediate rendering result) to the terminal device, but does not send the second rendering parameters to the terminal device. In the embodiment of FIG. 4, an example in which a rendered image of one frame is encoded is used for explanation.

[0216] S401: A server performs a rendering process on a three-dimensional scene based on rendering parameters to obtain a rendered image, where the rendering parameters include a first rendering parameter obtained from a terminal device.

[0217] S402: The server selects a first intermediate rendering result based on the intermediate rendering results generated in the rendering process.

[0218] S403: The server generates a virtual reference frame based on the first intermediate rendering result.

[0219] S404: The server predicts the rendered image based on the virtual reference frame to obtain a predicted image.

[0220] S405: The server encodes a residual image between the predicted image and the rendered image, and encodes the encoded data of the residual image into a bitstream.

[0221] For example, with regard to S401 to S405, please refer to the explanation of S201 to S205, and the details will not be explained again here.

[0222] S406: The server encodes the second intermediate rendering result, and encodes the encoded data of the second intermediate rendering result into a bitstream, where the second intermediate rendering result is a part of the first intermediate rendering result.

[0223] For example, S406 may be performed by the encoder of Fig. 1b. Specifically, the encoder encodes the second intermediate rendering result and encodes the encoded data of the second intermediate rendering result into a bitstream.

[0224] Fig. 5 is a diagram of an exemplary decoding process. Fig. 5 shows the decoding process corresponding to Fig. 4. In the embodiment of Fig. 5, the terminal device performs a rendering process based on locally generated rendering parameters to generate a third intermediate rendering result (the third intermediate rendering result and the second intermediate rendering result can form the first intermediate rendering result). In the embodiment of Fig. 5, an example in which one frame of image is decoded is used for explanation.

[0225] S501: A terminal device receives a bitstream.

[0226] S502: The terminal device parses the bitstream to obtain an analysis result, where the analysis result includes a residual image corresponding to the current frame and a second intermediate rendering result corresponding to the current frame.

[0227] For example, regarding S502, please refer to the description of S302, and the details will not be described again here.

[0228] For example, in the analysis step, encoded data of the second intermediate rendering result may be further extracted from the bitstream, and then the encoded data of the second intermediate rendering result may be input sequentially to an entropy decoding module, an inverse quantization module, and an inverse transform module to obtain the second intermediate rendering result.

[0229] It should be noted that if the server performs lossless compression on the second intermediate rendering result, the second intermediate rendering result obtained by the terminal device through analysis is the same as the second intermediate rendering result encoded by the encoder of the server, whereas if the server performs lossy compression on the second intermediate rendering result, the second intermediate rendering result obtained by the terminal device through analysis is different from the second intermediate rendering result encoded by the encoder of the server.

[0230] S503: The terminal device performs a rendering process on the three-dimensional scene based on rendering parameters corresponding to the current frame, and generates a third intermediate rendering result based on the second intermediate rendering result in the rendering process, where the rendering parameters include the first rendering parameters generated by the terminal device.

[0231] For example, in the rendering process, the second rendering module of the terminal device may generate a part of the first intermediate rendering result other than the second intermediate rendering result, i.e., a third intermediate rendering result. For details, please refer to the description of S303. The details will not be described again here. Then, the second rendering module may combine the second intermediate rendering result and the third intermediate rendering result to obtain the first intermediate rendering result.

[0232] It should be understood that since the third intermediate rendering result is generated by the terminal device, the type of the intermediate result included in the first intermediate rendering result obtained by the terminal device through combination is the same as the type of the intermediate result included in the first intermediate rendering result generated by the server, and the accuracy of the intermediate result included in the first intermediate rendering result obtained by the terminal device through combination is equal to or less than the accuracy of the intermediate result included in the first intermediate rendering result generated by the server.

[0233] S504: The terminal device generates a virtual reference frame based on the first intermediate rendering result, which may include the second intermediate rendering result and the third intermediate rendering result.

[0234] S505: The terminal device predicts the current frame based on the virtual reference frame to obtain a predicted image.

[0235] S506: The terminal device performs reconstruction based on the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0236] For example, with regard to S504 to S506, please refer to the explanation of S304 to S306, and the details will not be explained again here.

[0237] In this way, in the present application, partial rendering is performed by the terminal device, and the server only needs to send some intermediate rendering results to the terminal device. Therefore, in the present application, interaction delay can be reduced while ensuring that the bit rate overhead of the data stream sent by the server to the terminal device is effectively reduced. In addition, there is a strong correlation between the intermediate rendering results and the rendered image. Therefore, in the present application, the rendered image is encoded based on the intermediate rendering results, so that the image reconstruction quality can be guaranteed.

[0238] In the following, the encoding and decoding processes will be described using an example in which the first intermediate rendering result is a CGMV and the server sends a portion of the CGMV to the terminal device.

[0239] 6a is a diagram of an exemplary encoding process. In the embodiment of FIG. 6a, the server sends a part of the CGMV to the terminal device, and does not send the second rendering parameter to the terminal device. In the embodiment of FIG. 6a, an example in which one frame of rendered image is encoded is used for explanation.

[0240] S601: The server receives a first rendering parameter sent by a terminal device.

[0241] For example, the first rendering parameters may include camera parameters, such as, but not limited to, a view projection matrix, a camera dynamic range, a focal length, an aperture size, and a depth of field.

[0242] S602: The server generates second rendering parameters.

[0243] For example, the server can load scene geometry information (e.g., vertex coordinates of a triangular mesh), material data (e.g., material maps and material generation functions), texture data (e.g., texture maps), etc. that are pre-stored in a storage device (e.g., a hard disk or memory) to obtain second rendering parameters.

[0244] S603: The server performs a rendering process on the three-dimensional scene based on the first rendering parameter and the second rendering parameter to obtain a rendered image.

[0245] S604: The server selects a CGMV based on the intermediate rendering results generated in the rendering process.

[0246] For example, the server can input the first rendering parameters and the second rendering parameters to a graphics rendering engine (e.g., V-Ray, Unreal, or Unity), and the graphics rendering engine performs a rendering process to obtain a rendered image.

[0247] For example, in a rendering process, a graphics rendering engine can generate CGMV and other intermediate rendering results, and can select CGMV as the first intermediate rendering result. The CGMV may include CGMV of static objects and CGMV of dynamic objects.

[0248] For example, the process by which a graphics rendering engine generates a static CGMV may be as follows: for each point (which may include a vertex on a triangular mesh (also called a mesh)) and a non-vertex point on the triangular mesh (which will be called another point hereinafter (another point may be obtained by vertex interpolation)), a Model View Projection matrix M at time t1 (the current time) is calculated. t1 The projection of each point in (i.e., a sample of each vertex on the screen) is calculated and the projection is rasterized.

[0249] FIG. 6b is a diagram of an exemplary model-view projection transformation.

[0250] As shown in Figure 6b, for example, (0,0,0) in Figure 6b is the camera origin. +Y indicates the positive direction of the Y axis, +X indicates the positive direction of the X axis, and +Z indicates the positive direction of the Z axis.

[0251] For example, in a three-dimensional scene, points located on a triangular mesh between the near and far planes of a view frustum (where the camera origin is used as a vertex) may be projected onto the screen. In Figure 6b, points on a gray cube may be projected onto the screen. The parameters of the view frustum (e.g., horizontal, vertical, and diagonal angles of view) may be determined based on the camera parameters of the first rendering parameter. The model-view projection matrix M at time t1 t1 can be used to determine the corresponding samples onto which points on a triangle mesh located between the near and far planes of the view frustum (the camera origin is used as the vertex) are projected on the screen.

[0252] For example, the step of rasterizing the projection result may be as follows:

[0253] First, the array ZBuffer is pre-populated with samples corresponding to points on a triangle mesh between the near and far planes of the view frustum (the camera origin is used as a vertex). The initial value of each sample's position in the array ZBuffer is set to infinity (i.e., infinitely far from the camera).

[0254] Then, each sample [x,y] in the array ZBuffer is traversed, and the distance z from each sample to the camera, i.e., the distance from the point on the triangular mesh corresponding to each sample to the camera, is calculated. If the sample corresponds to a vertex on the triangular mesh, the corresponding distance can be calculated directly, or if the sample corresponds to another point on the triangular mesh, the corresponding distance can be obtained by performing interpolation based on the distance from the sample corresponding to the vertex to the camera. If z is less than the value of ZBuffer[x,y], the value of ZBuffer[x,y] is updated to the distance z of the point.

[0255] Next, the far surface distance value z of the camera parameters far and the near surface distance value z nearBased on this, the distance value corresponding to each sample in the array ZBuffer is normalized to obtain a depth value D whose variation range is between [0, 1], and all the depth values ​​D form a depth map. The normalization method is as follows:

number

[0256] Next, the view projection matrix M at time t2 (a rendering time corresponding to an arbitrary reconstructed image, where the graphics rendering engine does not necessarily perform rendering processing based on the image arrangement order in the video data, and therefore the encoder does not necessarily perform encoding based on the image arrangement order in the video data, and further, the reconstructed image may be a reconstructed image of an encoded image before the image to be rendered (the image that needs to be rendered at time t1) in the video data, or may be a reconstructed image of an encoded image after the image to be rendered) t2 and the current view projection matrix M t1 Based on the above, the CGMV of the static object between time t1 and time t2 is calculated and denoted as CGMV1. CGMV1=D * (M t1 -M t2 )

[0257] It should be understood that CGMVs of a static object between time t1 and N time points t2 can be generated to obtain N CGMVs. Each CGMV can correspond to one time point t2. N is a positive integer. For example, N is a positive integer less than 16. This is not a limitation in the present application.

[0258] For example, the process by which a graphics rendering engine generates dynamic CGMV may be as follows:

[0259] For example, a velocity buffer is maintained for each moving mesh (i.e., the triangular mesh of a dynamic object). The velocity buffer records the coordinates (x1, y1, z1) of each vertex on each moving mesh at time t1, and records the coordinates (x2, y2, z2) of each vertex on each moving mesh at time t2.

[0260] First, the motion vector P=(x1, y1, z1)-(x2, y2, z2) of each vertex on each moving mesh at time t1 and time t2 can be calculated.

[0261] Next, P is multiplied by the view projection matrix at time t1 to obtain the CGMV of the samples corresponding to each vertex on each moving mesh, which is denoted as CGMV2. CGMV2=P * (M t1 ).

[0262] Finally, the CGMV2 is rasterized. For details, please refer to the above description. We will not repeat the details here. In addition, interpolation can be performed on other points on the triangle mesh to obtain the CGMV2 of each sample.

[0263] It should be understood that CGMVs of a dynamic object between time t1 and N time points t2 can be generated to obtain N CGMVs, each of which can correspond to one time point t2.

[0264] For example, after the CGMV of the dynamic object and the CGMV of the static object are obtained, S605 and S608 may be executed separately.

[0265] S605: The server generates a virtual reference frame based on the CGMV and the reconstructed image.

[0266] For example, the type of CGMV is a motion vector type. If the type of the first intermediate rendering result is a motion vector type, a virtual reference frame is generated based on the first intermediate rendering result and the reconstructed image.

[0267] For example, the server can generate a virtual reference frame based on the CGMV of the dynamic object and the CGMV of the static object. For example, the server combines the CGMV of the dynamic object and the CGMV of the static object based on the sample to obtain a complete CGMV (i.e., a first intermediate rendering result), and then generates a virtual reference frame based on the complete CGMV. For example, the CGMV1 of the static object between time t1 and the i-th (i is a positive integer between 1 and N, inclusive) time t2 and the CGMV2 of the dynamic object between time t1 and the i-th time t2 may be combined to obtain a complete CGMV between time t1 and the i-th time t2 (i.e., the i-th complete CGMV). In this way, N complete CGMVs may be obtained, and N virtual reference frames may be generated.

[0268] For example, the process by which the server generates a virtual reference frame based on the i-th complete CGMV may be as follows:

[0269] First, a virtual reference frame can be initialized, whose size is the same as the size of the original reference frame.

[0270] Reconstructed image information is then obtained, including but not limited to luminance and chrominance information, bit depth, width, and height.

[0271] Then, a second sample in the i-th reconstructed image (i.e., the reconstructed image corresponding to the i-th time point t2) that corresponds to the first sample in the initialized virtual reference frame is determined based on the i-th complete CGMV, and the pixel value corresponding to the first sample in the initialized virtual reference frame is set based on the pixel value of the second sample in the i-th reconstructed image.

[0272] For example, for a first sample in a virtual reference frame, if it is determined based on the i-th complete CGMV that the position of the first sample in the i-th reconstructed image exceeds the boundary of the i-th reconstructed image, in one possible manner, the luminance value of the first sample may be set to a first default value, and the chrominance value may be set to a second default value. The first default value may be set to, for example, 16 based on requirements, and the second default value may be set to, for example, 128 based on requirements. This is not limited in the present application. In one possible manner, the pixel value of the first sample may be set to the average pixel value of the i-th reconstructed image.

[0273] For example, if it is determined based on the i-th complete CGMV that the position of the first sample in the i-th reconstructed image does not exceed the boundary of the i-th reconstructed image, the pixel value of the second sample corresponding to the first sample in the i-th reconstructed image is obtained, and this pixel value is used as the pixel value of the first sample.

[0274] S606: The server predicts the rendered image based on the virtual reference frame to obtain a predicted image.

[0275] S607: The server encodes a residual image between the predicted image and the rendered image, and encodes the encoded data of the residual image into a bitstream.

[0276] For example, if N=1, i.e., if there is one virtual reference frame, the one virtual reference frame may be used as a reference frame for inter-prediction to determine a predictive block that matches the block to be coded in the rendered image, and the residual block between the block to be coded and the predictive block is coded to obtain a bitstream.

[0277] For example, if N is greater than 1, that is, if there are multiple virtual reference frames, the multiple virtual reference frames can be used as candidate reference frames for inter-prediction to determine multiple predictive blocks that match the block to be coded in the rendered image, and one predictive block is determined correspondingly for one virtual reference frame. Then, to obtain a bitstream, the residual block between the optimal predictive block and the block to be coded is selected for coding.

[0278] For example, regardless of whether N is equal to or greater than 1, the virtual reference frame and the original reference frame may be used as candidate reference frames for inter prediction to determine multiple prediction blocks that match the current block in the rendered image. One prediction block is determined for each virtual reference frame / original reference frame accordingly. Then, to obtain a bitstream, the optimal prediction block and the residual block between the current block are selected for encoding.

[0279] For example, the bitstream description information may include a frame identifier. If the residual block is determined based on a prediction block determined based on an original reference frame and the current block to be coded, the frame identifier is the frame identifier of the original reference frame. If the residual block is determined based on a prediction block determined based on a virtual reference frame and the current block to be coded, the frame identifier is the frame identifier of the virtual reference frame.

[0280] S608: The server encodes the first CGMV and encodes the encoded data of the first CGMV into a bitstream.

[0281] For example, after generating a complete CGMV, the server can encode a portion of the CGMV. It should be understood that the server generates N complete CGMVs. If the predictive block used to calculate the residual block is a block in the i-th virtual reference frame, the portion of the complete CGMV used to determine the i-th virtual reference frame may be encoded. The portion of the CGMV may be referred to as a first CGMV, i.e., a second intermediate rendering result.

[0282] In one possible manner, the first CGMV is a CGMV of a static object, and the server can further send the encoded CGMV of the static object to the terminal device, in this way, the terminal device does not need to calculate the CGMV of the static object, thereby saving the computational power of the terminal device.

[0283] In one possible manner, the first CGMV is a CGMV of a dynamic object, and the server can further send the encoded CGMV of the dynamic object to the terminal device. In this way, the terminal device does not need to calculate the CGMV of the dynamic object, thereby saving the calculation power of the terminal device. In addition, the calculation amount of the CGMV of the dynamic object is greater than the calculation amount of the CGMV of the static object. Therefore, in order to further save the calculation power of the terminal device, the CGMV of the dynamic object is encoded and sent to the terminal device.

[0284] For example, the CGMV of a dynamic object may include the CGMV of a rigid dynamic object and / or the CGMV of a non-rigid dynamic object.

[0285] In one possible approach, the first CGMV may include a CGMV of a rigid dynamic object and a CGMV of a non-rigid dynamic object.

[0286] In one possible manner, the first CGMV may include a CGMV of a rigid dynamic object, and the server may further transmit the encoded CGMV of the rigid dynamic object to the terminal device, thereby reducing the bit rate compared to encoding the CGMV of a rigid dynamic object and the CGMV of a non-rigid dynamic object.

[0287] In one possible manner, the first CGMV may include a CGMV of a non-rigid dynamic object, and the server may further transmit the encoded CGMV of the non-rigid dynamic object to the terminal device. In this way, the bit rate can be reduced compared to encoding the CGMV of a rigid dynamic object and the CGMV of a non-rigid dynamic object. In addition, since the number of triangular meshes of a non-rigid dynamic object is large, the calculation amount of the CGMV of the non-rigid dynamic object is large. Therefore, compared with encoding the CGMV of a rigid dynamic object, encoding the CGMV of a non-rigid dynamic object can save more calculation power of the terminal device.

[0288] For example, in the embodiment of Fig. 6a, the bitstream description information may further include a first instruction identifier and / or a fourth instruction identifier, where the first instruction identifier may indicate the type and subtype of the second intermediate rendering result, and the fourth instruction identifier may indicate whether the bitstream includes encoded data of the second intermediate rendering result.

[0289] For example, the motion vector types may include subtype A1, subtype A2, and subtype A3. The subtype corresponding to the CGMV of a static object is subtype A1, the subtype corresponding to the CGMV of a rigid dynamic object is subtype A2, and the subtype corresponding to the CGMV of a non-rigid dynamic object is subtype A3. In the embodiment of FIG. 6a, the first indication identifier can indicate the type and subtype of the first CGMV. In this way, the terminal device can know the specific type of the second CGMV to be generated (the second CGMV is a part of a CGMV other than the first CGMV).

[0290] For example, the motion vector type is represented by V, subtype A1 is represented by 1, subtype A2 is represented by 2, and subtype A3 is represented by 3. If the first CGMV is a CGMV of a non-rigid dynamic object, the first designation identifier may be V3.

[0291] For example, if the fourth instruction identifier is first default information (e.g., 1), it indicates that the bitstream contains encoded data of the second intermediate rendering result, or if the fourth instruction identifier is second default information (e.g., 0), it indicates that the bitstream does not contain encoded data of the second intermediate rendering result.

[0292] In the following, a decoding step corresponding to the encoding step of FIG. 6a will be described using an example in which the bitstream description information of the bitstream includes a first indication identifier and a fourth indication identifier.

[0293] Fig. 7 is a diagram of an exemplary decoding process. Fig. 7 shows the decoding process corresponding to Fig. 6a. In the embodiment of Fig. 7, the terminal device performs a rendering process based on locally generated rendering parameters to generate other parts of the CGMV. In the embodiment of Fig. 7, an example in which one frame of image is decoded is used for explanation.

[0294] S701: A terminal device receives a bitstream.

[0295] S702: The terminal device parses the bitstream to obtain a parsing result, where the parsing result includes a residual image corresponding to the current frame and a first CGMV corresponding to the current frame.

[0296] For example, for S701 and S702, please refer to the description of S501 and S502, and the details will not be described again here.

[0297] S703: The terminal device generates a first rendering parameter and a fifth rendering parameter.

[0298] For example, the first rendering parameters may include camera parameters, such as, but not limited to, a view projection matrix, a camera dynamic range, a focal length, an aperture size, and a depth of field.

[0299] For example, the terminal device can load scene geometry information (e.g., vertex coordinates of a triangular mesh), material data (e.g., material maps and material generation functions), texture data (e.g., texture maps), etc. that are pre-stored in a storage device (e.g., a hard disk or memory) to obtain the fifth rendering parameter.

[0300] S704: The terminal device performs a rendering process on the three-dimensional scene based on the first rendering parameter and the fifth rendering parameter, and generates a second CGMV based on the first CGMV in the rendering process.

[0301] For example, the terminal device may input the first rendering parameter and the fifth rendering parameter to a graphics rendering engine (e.g., V-Ray, Unreal, or Unity), which performs a rendering process on the three-dimensional scene. In the rendering process, a second CGMV is generated based on the first CGMV.

[0302] For example, the analysis result may further include a first instruction identifier and a fourth instruction identifier. If the terminal device determines based on the fourth instruction identifier that the bitstream includes encoded data of a second intermediate rendering result, the terminal device can determine the type and subtype of the second intermediate rendering result based on the first instruction identifier. Then, in the rendering process, a third intermediate rendering result of a different subtype is generated based on the type and subtype of the second intermediate rendering result. The different subtype is one of the subtypes included in the type corresponding to the second intermediate rendering result, other than the subtype of the second intermediate rendering result.

[0303] 7, if the terminal device determines based on the first instruction identifier that the type of the first CGMV is a motion vector type and the subtype is subtype A1, i.e., the first CGMV sent by the server is a CGMV of a static object, then in the rendering process, the graphic rendering engine of the terminal device does not need to generate a CGMV of a static object, but only needs to generate a CGMV of a dynamic object (in this case, other subtypes include subtype A2 and subtype A3), i.e., the second CGMV is a CGMV of a dynamic object.

[0304] 7, if the terminal device determines based on the first instruction identifier that the type of the first CGMV is a motion vector type and the subtypes are subtype A2 and subtype A3, i.e., the first CGMV sent by the server is a CGMV of a rigid dynamic object and a CGMV of a non-rigid dynamic object, then in the rendering processing step, the graphic rendering engine of the terminal device does not need to generate a CGMV of a rigid dynamic object and a CGMV of a non-rigid dynamic object, but only needs to generate a CGMV of a static object (in this case, another subtype includes subtype A1), i.e., the second CGMV is a CGMV of a static object.

[0305] 7, if the terminal device determines based on the first instruction identifier that the type of the first CGMV is a motion vector type and the subtype is subtype A2, i.e., the first CGMV sent by the server is a CGMV of a rigid dynamic object, then in the rendering processing step, the graphic rendering engine of the terminal device does not need to generate a CGMV of a rigid dynamic object, but only needs to generate a CGMV of a static object and a CGMV of a non-rigid dynamic object (in this case, other subtypes include subtype A1 and subtype A3). That is, the second CGMV is a CGMV of a static object and a CGMV of a non-rigid dynamic object.

[0306] 7, if the terminal device determines based on the first instruction identifier that the type of the first CGMV is a motion vector type and the subtype is subtype A3, i.e., the first CGMV sent by the server is a CGMV of a non-rigid dynamic object, then in the rendering process, the graphic rendering engine of the terminal device does not need to generate a CGMV of a non-rigid dynamic object, but only needs to generate a CGMV of a static object and a CGMV of a rigid dynamic object (in this case, other subtypes include subtype A1 and subtype A2). That is, the second CGMV is a CGMV of a static object and a CGMV of a rigid dynamic object.

[0307] It should be understood that the process of the terminal device generating the CGMV of the dynamic object / CGMV of the static object should be referred to the above description, and the details will not be described again here.

[0308] For example, the analysis result may further include a frame identifier. If the terminal device can determine that the frame identifier is the frame identifier of the virtual reference frame, the terminal device can determine the time instant of the model-view projection matrix that needs to be used to generate the second CGMV.

[0309] S705: The terminal device generates a virtual reference frame based on the first CGMV and the second CGMV.

[0310] For example, the terminal device can combine the first CGMV and the second CGMV, and then generate a virtual reference frame based on the combined CGMV (i.e., the complete CGMV). For details, please refer to the above description. The details will not be described again here.

[0311] S706: The terminal device predicts the current frame based on the virtual reference frame to obtain a predicted image.

[0312] S707: The terminal device performs reconstruction based on the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0313] For example, see the above description of S706 and S707, and the details will not be described again here.

[0314] Because the CGMV generated by the server is more accurate than the CGMV generated by the terminal device, the CGMV used by the terminal device to generate a virtual reference frame can be more accurate, thereby improving the accuracy of the virtual reference frame and the image quality of the reconstructed image.In addition, since the terminal device only needs to generate a portion of the CGMV, the computing power of the terminal device can be saved and the decoding efficiency can be improved.In addition, compared with the prior art in which the server transmits all the CGMV for the same quality, the server in this application only transmits some of the CGMV, thereby reducing the bit rate overhead of the data stream transmitted by the server to the terminal device.

[0315] In the following, the encoding and decoding processes are described using an example in which the first intermediate rendering result is an intermediate rendered image and the server sends a portion of an area image within the intermediate rendered image to the terminal device.

[0316] 8 is a diagram of an exemplary encoding process. In the embodiment of FIG. 8, the server transmits a part of the area image in the intermediate rendered image to the terminal device, and does not transmit the second rendering parameters to the terminal device. In the embodiment of FIG. 8, an example in which one frame of the rendered image is encoded is used for explanation.

[0317] S801: The server receives a first rendering parameter sent by a terminal device.

[0318] S802: The server generates second rendering parameters.

[0319] For example, for S801 and S802, please refer to the description of S601 and S602, and the details will not be described again here.

[0320] S803: The server performs a rendering process on the three-dimensional scene based on the first rendering parameter and the second rendering parameter to obtain a rendered image.

[0321] S804: The server selects an intermediate rendered image based on the intermediate rendering results generated in the rendering process.

[0322] In one possible approach, the intermediate rendered image may be an image generated by performing some rendering work with a graphics rendering engine (the rendered image is an image generated by performing all rendering work with the server's graphics rendering engine).

[0323] In one possible approach, the intermediate rendered image may be an image generated by performing rendering by a graphics rendering engine of the server based on low-precision rendering parameters or some rendering parameters (the rendered image is an image generated by performing rendering by a graphics rendering engine based on all high-precision rendering parameters).

[0324] For example, intermediate rendered images may include, but are not limited to, intermediate rendered images without indirect lighting rendering, intermediate rendered images without specular processing, intermediate rendered images without highlight processing, etc. This is not a limitation of this application.

[0325] S805: The server uses the intermediate rendered image as a virtual reference frame.

[0326] For example, the server can use an intermediate rendered image as a virtual frame of reference.

[0327] S806: Predict the rendered image based on the virtual reference frame to obtain a predicted image.

[0328] S807: A residual image between the predicted image and the rendered image is encoded, and the encoded data of the residual image is encoded into a bitstream.

[0329] For example, see the above description of S806 and S807, and the details will not be repeated here.

[0330] S808: The server encodes the first area image in the intermediate rendered image, and encodes the encoded data of the first area image into a bitstream.

[0331] For example, after generating a complete intermediate rendered image, the server can encode a first area image within the intermediate rendered image.

[0332] In one possible manner, the first area image may be an image of an area corresponding to four corners in the intermediate rendered image, and the image size of the area corresponding to each corner may be set based on requirements, which is not limited in this application.

[0333] In one possible way, the first area image may be an image of an area other than the areas corresponding to the four corners in the intermediate rendered image.

[0334] It should be understood that the first area image may instead be an image of another area within the intermediate rendered image, which is not a limitation in this application.

[0335] For example, in the embodiment of Figure 8, the bitstream description information may further include a first instruction identifier and / or a fourth instruction identifier, where the first instruction identifier may indicate the type and subtype of the second intermediate rendering result, and the fourth instruction identifier may indicate whether the bitstream includes encoded data of the second intermediate rendering result.

[0336] For example, the image type may include subtype B1 and subtype B2. For example, a subtype corresponding to an area corresponding to the four corners in the intermediate rendered image is subtype B1, and a subtype corresponding to an area other than the area corresponding to the four corners in the intermediate rendered image is subtype B2. In the embodiment of FIG. 8, the first designation identifier may designate the type and subtype of the first area image. In this way, the terminal device may know the specific type and specific area of ​​the image to be generated, i.e., the second area image (the second area image is an image other than the first area image in the intermediate rendered image).

[0337] For example, the image type is represented by M, the subtype B1 is represented by 1, and the subtype B2 is represented by 2. Thus, if the first area image is an image of an area corresponding to the four corners in the intermediate rendered image, the first designation identifier may be M1.

[0338] In the following, a decoding process corresponding to the encoding process of FIG. 8 will be described using an example in which the bitstream description information of the bitstream includes a first indication identifier and a fourth indication identifier.

[0339] Fig. 9 is a diagram of an exemplary decoding process. Fig. 9 shows the decoding process corresponding to Fig. 8. In the embodiment of Fig. 9, a terminal device performs a rendering process based on locally generated rendering parameters to generate a part of an area image in an intermediate rendered image. In the embodiment of Fig. 9, an example in which one frame of image is decoded is used for explanation.

[0340] S901: A terminal device receives a bitstream.

[0341] S902: The terminal device parses the bitstream to obtain an analytic result, where the analytic result includes a residual image corresponding to the current frame and a first area image corresponding to the current frame.

[0342] For example, for S901 and S902, please refer to the description of S501 and S502, and the details will not be described again here.

[0343] S903: The terminal device generates a first rendering parameter and a fifth rendering parameter.

[0344] For example, regarding S903, please refer to the description of S703, and the details will not be described again here.

[0345] S904: The terminal device performs a rendering process based on the first rendering parameter and the fifth rendering parameter, and generates a second area image based on the first area image in the rendering process.

[0346] For example, the terminal device can input the first rendering parameter and the fifth rendering parameter to a graphics rendering engine (e.g., V-Ray, Unreal, or Unity), and the graphics rendering engine performs a rendering process on the three-dimensional scene. In the rendering process, a second area image is generated based on the first area image.

[0347] For example, the analysis result may further include a first instruction identifier and a fourth instruction identifier. If the terminal device determines based on the fourth instruction identifier that the bitstream includes encoded data of a second intermediate rendering result, the terminal device can determine the type and subtype of the second intermediate rendering result based on the first instruction identifier. Then, in the rendering process, a third intermediate rendering result of a different subtype is generated based on the type and subtype of the second intermediate rendering result. The different subtype is one of the subtypes included in the type corresponding to the second intermediate rendering result, other than the subtype of the second intermediate rendering result.

[0348] 9, if the terminal device determines based on the first instruction identifier that the type of the first area image is an image type and the subtype is subtype B1, i.e., the first area image sent by the server is an image of an area corresponding to the four corners in the intermediate rendered image, then in the rendering process, the graphics rendering engine of the terminal device does not need to generate an image of the area corresponding to the four corners, but only needs to generate an image of an area other than the area corresponding to the four corners (in this case, the other subtype is subtype B2). That is, the second area image is an image of an area other than the area corresponding to the four corners.

[0349] 9, if the terminal device determines based on the first instruction identifier that the type of the first area image is an image type and the subtype is subtype B2, i.e., the first area image sent by the server is an image of an area other than the areas corresponding to the four corners in the intermediate rendered image, then in the rendering process, the graphics rendering engine of the terminal device does not need to generate images of areas other than the areas corresponding to the four corners, but only needs to generate images of the areas of the four corners (in this case, the other subtype is subtype B1). That is, the second area image is an image of the area corresponding to the four corners.

[0350] S905: The terminal device uses the intermediate rendered image obtained by combining the first area image and the second area image as a virtual reference frame.

[0351] For example, the first area image and the second area image may be combined on a pixel basis to obtain an intermediate rendered image, and then the intermediate rendered image obtained by the combination is used as a virtual reference frame.

[0352] S906: The terminal device predicts the current frame based on the virtual reference frame to obtain a predicted image.

[0353] S907: The terminal device performs reconstruction based on the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0354] For example, S906 and S907, please refer to the above description, and the details will not be described again here.

[0355] Because the intermediate rendered image generated by the server is more accurate than the intermediate rendered image generated by the terminal device, the intermediate rendered image used by the terminal device to generate a virtual reference frame can be more accurate, thereby improving the accuracy of the virtual reference frame and the image quality of the reconstructed image.In addition, the terminal device only needs to generate a portion of the area image in the intermediate rendered image, which can save the computing power of the terminal device and improve decoding efficiency.In addition, compared with the prior art in which the server transmits all CGMVs for the same quality, the server in this application transmits only a portion of the area image in the intermediate rendered image, which can reduce the bit rate overhead of the data stream transmitted by the server to the terminal device.In addition, because the difference between the intermediate rendered image and the rendered image is small, a small residual can be obtained, and the bit rate of the encoding data of the residual block can be further reduced.

[0356] It should be noted that if the bitstream description parameters of the bitstream do not include the fourth instruction identifier, the terminal device of the embodiments of Figures 7 and 9 can determine whether the bitstream includes encoded data of the second intermediate rendering result based on pre-agreed information. If the bitstream description parameters of the bitstream do not include the first instruction identifier, the terminal device of the embodiments of Figures 7 and 9 can determine the type and subtype of the second intermediate rendering result in the analysis result based on pre-agreed information.

[0357] It should be understood that even if the first intermediate rendering result is other information, encoding and decoding may be performed with reference to the above-mentioned scheme, and whether to transmit all / part of the other information to the terminal device may be determined based on the data amount of the other information, which is not limited in the present application.

[0358] For example, if the first intermediate rendering result of the server includes CGMV and an intermediate rendered image, in the process of encoding the rendered image, a virtual reference frame generated based on the CGMV and a virtual reference frame generated based on the intermediate rendered image may be used as candidate reference frames, multiple predictive blocks matching the encoding target block in the rendered image may be determined based on the candidate reference frames, an optimal predictive block may be selected, and a residual between the optimal predictive block and the encoding target block may be calculated to determine a residual block. In this case, a frame identifier used to determine the virtual reference frame corresponding to the optimal residual block may be coded into the bitstream, and a value of the first indication identifier may be of the type of the first intermediate rendering result and set to the identifier used to determine the virtual reference frame corresponding to the optimal residual block.

[0359] It should be understood that if the first intermediate rendering result further includes other information, if the virtual reference frame generated based on the other information, the virtual reference frame generated based on CGMV, and the virtual reference frame generated based on the intermediate rendered image are all used as candidate reference frames, the optimal residual block may be selected for encoding in the above manner, the corresponding frame identifier is coded into the bitstream, and the value of the first indication identifier is set, which is not limited in the present application.

[0360] For example, if the first rendering parameters generated by the terminal device are part of the parameters required for the rendering process by the graphics rendering engine, the server can generate second rendering parameters. Since the rendering parameters generated by the server are more accurate than the rendering parameters generated by the terminal device, the server can transmit some or all of the second rendering parameters to the terminal device. In this way, the first intermediate rendering result generated by the terminal device can be more accurate, the accuracy of the virtual reference frame can be improved, and the image quality of the image obtained by decoding based on the virtual reference frame can be improved.

[0361] Fig. 10 is a diagram of an exemplary encoding process. In the embodiment of Fig. 10, the server does not send the first intermediate rendering result to the terminal device, but sends the third rendering parameters (part or all of the second rendering parameters) to the terminal device. In the embodiment of Fig. 10, an example in which a rendered image of one frame is encoded is used for explanation.

[0362] S1001: The server receives a first rendering parameter sent by a terminal device.

[0363] For example, the first rendering parameters may include camera parameters, such as, but not limited to, a view projection matrix, a camera dynamic range, a focal length, an aperture size, and a depth of field.

[0364] S1002: The server generates second rendering parameters.

[0365] For example, the server can load scene geometry information (e.g., vertex coordinates of a triangular mesh), material data (e.g., material maps and material generation functions), texture data (e.g., texture maps), motion information of dynamic objects, etc., that are pre-stored in a storage device (e.g., a hard disk or memory) to obtain second rendering parameters.

[0366] For example, the dynamic object motion information may include motion information (eg, motion tracks) of rigid dynamic objects and / or motion information (eg, motion tracks) of non-rigid dynamic objects.

[0367] S1003: The server performs a rendering process on the three-dimensional scene based on the first rendering parameter and the second rendering parameter to obtain a rendered image.

[0368] S1004: The server selects a first intermediate rendering result based on the intermediate rendering results generated in the rendering process.

[0369] For example, in the embodiment of FIG. 10, the first intermediate rendering result may be a CGMV or may be an intermediate rendered image.

[0370] S1005: The server generates a virtual reference frame based on the first intermediate rendering result.

[0371] S1006: The server predicts the rendered image based on the virtual reference frame to obtain a predicted image.

[0372] S1007: The server encodes a residual image between the predicted image and the rendered image, and encodes the encoded data of the residual image into a bitstream.

[0373] For example, with regard to S1005 to S1007, please refer to the description of the above embodiment, and the details will not be described again here.

[0374] S1008: The server encodes the third rendering parameter into a bitstream.

[0375] For example, the third rendering parameter may be a part of the second rendering parameter, or may be all of the second rendering parameter, which is not limited in the present application. Compared with encoding all of the second rendering parameters into the bitstream, encoding only a part of the second rendering parameter into the bitstream can further reduce the bitrate overhead of the data stream transmitted by the server to the terminal device.

[0376] For example, because the data amount of the third rendering parameter is small, the server may encode the third rendering parameter, encode the encoded data of the third rendering parameter into a bitstream, and transmit the bitstream to the server, or the server may not encode the third rendering parameter, but may directly encode the third rendering parameter into the bitstream, which is not limited in the present application.

[0377] In one possible manner, the third rendering parameter may include motion information of rigid motion objects and motion information of non-rigid dynamic objects, in other words, the motion information of rigid motion objects and motion information of non-rigid dynamic objects are coded into the bitstream.

[0378] In one possible manner, the third rendering parameter may include motion information of a rigid motion object. In other words, the motion information of the rigid motion object is encoded into the bitstream. In this way, by encoding the motion information of the rigid motion object into the bitstream, the bitrate overhead of the data stream transmitted by the server to the terminal device can be further reduced compared to encoding the motion information of the rigid motion object and the motion information of the non-rigid dynamic object into the bitstream. In one possible manner, the third rendering parameter may include motion information of a non-rigid motion object. In other words, the motion information of the non-rigid dynamic object is encoded into the bitstream. In this way, by encoding the motion information of the non-rigid dynamic object into the bitstream, the bitrate overhead of the data stream transmitted by the server to the terminal device can be further reduced compared to encoding the motion information of the rigid motion object and the motion information of the non-rigid dynamic object into the bitstream.

[0379] In this way, the data amount of the third rendering parameters is much less than the data amount of the second intermediate rendering result, and transmitting the third rendering parameters can further reduce the bit rate compared to transmitting the second intermediate rendering result.

[0380] For example, in the embodiment of Figure 10, the bitstream description information may include at least one of the following: a first instruction identifier, a second instruction identifier, a third instruction identifier, or a fourth instruction identifier: the first instruction identifier indicates whether the bitstream includes coded data of a first intermediate rendering result, the second instruction identifier may indicate the type of the first intermediate rendering result, the third instruction identifier may indicate whether the bitstream includes third rendering parameters, and the fourth instruction identifier may indicate the type of the third rendering parameters.

[0381] For example, in the embodiment of Figure 10, the first instruction identifier may be second predetermined information indicating that the bitstream does not include encoded data of the first intermediate rendering result. It should be understood that the first instruction identifier in Figure 10 and the fourth instruction identifier in the embodiment of Figure 6 may be the same instruction identifier. If the fourth instruction identifier is the second predetermined information, it can be determined that the bitstream does not include encoded data of the first intermediate rendering result.

[0382] For example, the first intermediate rendering result may correspond to one or more types. If the first intermediate rendering result corresponds to one type, the type of the first intermediate rendering result may be used as the value of the second indication identifier and encoded into the bitstream. If the first intermediate rendering result corresponds to multiple types, the type of the first intermediate rendering result used to generate a virtual reference frame of the optimal prediction block may be used as the value of the second indication identifier and encoded into the bitstream.

[0383] For example, if the third instruction identifier is first default information (e.g., 1), it indicates that the bitstream includes the third rendering parameter, or if the third instruction identifier is second default information (e.g., 0), it indicates that the bitstream does not include the third rendering parameter.

[0384] For example, the parameters included in the second rendering parameters may correspond to multiple types. For example, a type corresponding to motion information of a rigid dynamic object is type C1, and a type corresponding to motion information of a non-rigid dynamic object is type C2. The type of the third rendering parameter coded into the bitstream may be used as a value of a fourth indication identifier and coded into the bitstream.

[0385] In the following, the decoding process corresponding to the encoding process of Figure 10 is described using an example in which the bitstream description information of the bitstream includes a first instruction identifier, a second instruction identifier, a third instruction identifier, and a fourth instruction identifier.

[0386] Fig. 11 is a diagram of an exemplary decoding process. Fig. 11 shows the decoding process corresponding to Fig. 10. In the embodiment of Fig. 11, the terminal device performs a rendering process based on first rendering parameters generated locally and second rendering parameters received from the server to generate a first intermediate rendering result. The third rendering parameters include a part of the second rendering parameters. In the embodiment of Fig. 11, an example in which one frame of image is decoded is used for explanation.

[0387] S1101: A terminal device receives a bitstream.

[0388] S1102: The terminal device parses the bitstream to obtain an analysis result, where the analysis result includes a residual image corresponding to the current frame and a third rendering parameter corresponding to the current frame.

[0389] S1103: The terminal device generates a first rendering parameter and a fourth rendering parameter.

[0390] For example, the first rendering parameters may include camera parameters, such as, but not limited to, a view projection matrix, a camera dynamic range, a focal length, an aperture size, and a depth of field.

[0391] For example, the terminal device can obtain a third instruction identifier from the bitstream through analysis. If the terminal device determines that the bitstream includes a third rendering parameter based on the third instruction identifier, the terminal device can determine the type of the third rendering parameter based on the fourth instruction identifier, and then generate the fourth rendering parameter based on the type of the third rendering parameter.

[0392] For example, if the server encodes a part of the second rendering parameters into the bitstream, i.e., if the third rendering parameters include a part of the second rendering parameters, the fourth instruction identifier includes the type of the part of the second rendering parameters. In this case, the terminal device can generate the fourth rendering parameters based on the type of the third rendering parameters. The fourth rendering parameters are parameters of the second rendering parameters other than the third rendering parameters.

[0393] For example, if the type of the third rendering parameter is type C1, i.e., if the third rendering parameter is motion information of a rigid dynamic object, the fourth rendering parameter generated by the terminal device may include, for example, scene geometry information (e.g., vertex coordinates of a triangular mesh), material data (e.g., material maps and material generation functions), texture data (e.g., texture maps), and motion information of a non-rigid dynamic object.

[0394] For example, if the type of the third rendering parameter is type C2, i.e., if the third rendering parameter is motion information of a non-rigid dynamic object, the fourth rendering parameter generated by the terminal device may include, for example, scene geometry information (e.g., vertex coordinates of a triangular mesh), material data (e.g., material maps and material generation functions), texture data (e.g., texture maps), and motion information of a rigid dynamic object.

[0395] For example, if the type of the third rendering parameters includes type C1 and type C2, i.e., if the third rendering parameters include motion information of rigid dynamic objects and motion information of non-rigid dynamic objects, the fourth rendering parameters generated by the terminal device may include, for example, scene geometry information (e.g., vertex coordinates of a triangular mesh), material data (e.g., a material map and a material generation function), and texture data (e.g., a texture map).

[0396] S1104: The terminal device performs a rendering process on the three-dimensional scene based on the first rendering parameter, the third rendering parameter, and the fourth rendering parameter, and generates a first intermediate rendering result in the rendering process.

[0397] For example, if the third rendering parameters include some of the parameters of the second rendering parameters, a rendering process is performed on the three-dimensional scene based on the first rendering parameters, the third rendering parameters, and the fourth rendering parameters.

[0398] It should be understood that if the server encodes all of the second rendering parameters into the bitstream, i.e., the third rendering parameters include all of the parameters of the second rendering parameters, the fourth instruction identifier includes all types of the second rendering parameters, and the terminal device can obtain all of the second rendering parameters by decoding. In this case, the terminal device does not need to generate the fourth rendering parameters, and then performs rendering processing on the three-dimensional scene based on the first rendering parameters and the third rendering parameters.

[0399] For example, the first intermediate rendering result may be a CGMV or intermediate rendered image.

[0400] For example, if it is determined based on a first instruction identifier obtained from the bitstream by analysis that the bitstream does not contain encoded data of a first intermediate rendering result, in the rendering processing step, the terminal device can generate a first intermediate rendering result of a corresponding type based on a second instruction identifier obtained by analysis.

[0401] S1105: A virtual reference frame is generated based on the first intermediate rendering result.

[0402] S1106: Predict the current frame based on the virtual reference frame to obtain a predicted image.

[0403] S1107: Perform reconstruction based on the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0404] For example, S1106 and S1107, please refer to the above description, and the details will not be described again here.

[0405] It should be noted that if the bitstream description parameters of the bitstream do not include the first indication identifier, the terminal device of the embodiment of FIG. 11 can determine whether the bitstream includes coded data of the first intermediate rendering result based on pre-agreed information. If the bitstream description parameters of the bitstream do not include the second indication identifier, the terminal device of the embodiment of FIG. 11 can determine the type of the first intermediate rendering result based on pre-agreed information. If the bitstream description parameters of the bitstream do not include the third indication identifier, the terminal device of the embodiment of FIG. 11 can determine whether the bitstream includes the third rendering parameter based on pre-agreed information. If the bitstream description parameters of the bitstream do not include the fourth indication identifier, the terminal device of the embodiment of FIG. 11 can determine the type of the third rendering parameter of the analysis result based on pre-agreed information.

[0406] For example, the data amount of the second rendering parameters is small (several to several tens of KB). Therefore, even if the server further transmits all or part of the second rendering parameters to the terminal device in addition to the second intermediate rendering result, the bit rate overhead of the data stream transmitted by the server to the terminal device is smaller than the bit rate overhead of the data stream transmitted by the server to the terminal device in the prior art, which can further save the computing power of the terminal device.

[0407] 12 is a diagram of an exemplary encoding process. In the embodiment of FIG. 12, the server sends the second intermediate rendering result and the third rendering parameters to the terminal device. In the embodiment of FIG. 12, an example in which one frame of a rendered image is encoded is used for explanation.

[0408] S1201: The server receives a first rendering parameter sent by a terminal device.

[0409] For example, the first rendering parameters may include camera parameters and motion information of non-rigid dynamic objects (e.g., skeletal nodes of a moving roll). For example, the camera parameters may include, but are not limited to, parameters such as a view projection matrix, a camera dynamic range, a focal length, an aperture size, and a depth of field.

[0410] S1202: The server generates second rendering parameters.

[0411] For example, the server can load scene geometry information (e.g., vertex coordinates of a triangular mesh), material data (e.g., material maps and material generation functions), texture data (e.g., texture maps), motion information of dynamic objects, etc., that are pre-stored in a storage device (e.g., a hard disk or memory) to obtain second rendering parameters.

[0412] For example, the motion information of the dynamic object may include motion information (e.g., motion tracks) of a rigid dynamic object and / or motion information (e.g., motion tracks) of a non-rigid dynamic object (i.e., a non-rigid dynamic object other than a moving roll).

[0413] S1203: The server performs a rendering process on the three-dimensional scene based on the first rendering parameter and the second rendering parameter to obtain a rendered image.

[0414] S1204: The server selects a first intermediate rendering result based on the intermediate rendering results generated in the rendering process.

[0415] For example, in the embodiment of FIG. 12, the first intermediate rendering result may be a CGMV or may be an intermediate rendered image.

[0416] S1205: The server generates a virtual reference frame based on the first intermediate rendering result.

[0417] S1206: The server predicts the rendered image based on the virtual reference frame to obtain a predicted image.

[0418] S1207: The server encodes a residual image between the predicted image and the rendered image, and encodes the encoded data of the residual image into a bitstream.

[0419] S1208: The server encodes the second intermediate rendering result, and encodes the encoded data of the second intermediate rendering result into a bitstream, where the second intermediate rendering result is a part of the first intermediate rendering result.

[0420] S1209: The server encodes the third rendering parameter into a bitstream.

[0421] For example, regarding S1205 to S1209, please refer to the description of the above embodiment, and the details will not be described again here.

[0422] For example, in the embodiment of Figure 12, the bitstream description information may include at least one of the following: a first instruction identifier, a second instruction identifier, a third instruction identifier, and a fourth instruction identifier: the first instruction identifier may indicate a type and subtype of the second intermediate rendering result, the second instruction identifier may indicate whether the bitstream includes third rendering parameters, the third instruction identifier may indicate a type of the third rendering parameters, and the fourth instruction identifier may indicate whether the bitstream includes coded data of the second intermediate rendering result.

[0423] In the following, the decoding process corresponding to the encoding process of Figure 12 is described using an example in which the bitstream description information of the bitstream includes a first instruction identifier, a second instruction identifier, a third instruction identifier, and a fourth instruction identifier.

[0424] Fig. 13 is a diagram of an exemplary decoding process. Fig. 13 shows the decoding process corresponding to Fig. 12. In the embodiment of Fig. 13, the terminal device performs a rendering process based on the first rendering parameters generated locally and the third rendering parameters received from the server to generate a third intermediate rendering result. The third rendering parameters include a part of the second rendering parameters. In the embodiment of Fig. 13, an example in which one frame of image is decoded is used for explanation.

[0425] S1301: A terminal device receives a bitstream.

[0426] S1302: The terminal device parses the bitstream to obtain an analysis result, which includes a residual image corresponding to the current frame, a second intermediate rendering result corresponding to the current frame, and a third rendering parameter corresponding to the current frame.

[0427] S1303: The terminal device generates a first rendering parameter and a fourth rendering parameter.

[0428] For example, the first rendering parameters include camera parameters and motion information of non-rigid dynamic objects (e.g., skeletal nodes of a moving roll). For example, the camera parameters may include, but are not limited to, parameters such as a view projection matrix, a camera dynamic range, a focal length, an aperture size, and a depth of field.

[0429] S1304: The terminal device performs a rendering process on the three-dimensional scene based on the first rendering parameter, the third rendering parameter, and the fourth rendering parameter, and generates a third intermediate rendering result based on the second intermediate rendering result in the rendering process.

[0430] S1305: The terminal device generates a virtual reference frame based on the second intermediate rendering result and the third intermediate rendering result.

[0431] S1306: The terminal device predicts the current frame based on the virtual reference frame to obtain a predicted image.

[0432] S1307: The terminal device performs reconstruction based on the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0433] For example, see the above description of S1304 and S1307, and the details will not be repeated here.

[0434] It should be noted that if the bitstream description parameters of the bitstream do not include the fourth instruction identifier, the terminal device of the embodiment of FIG. 13 can determine whether the bitstream includes coded data of the second intermediate rendering result based on pre-agreed information. If the bitstream description parameters of the bitstream do not include the first instruction identifier, the terminal device of the embodiment of FIG. 13 can determine the type and subtype of the second intermediate rendering result based on pre-agreed information. If the bitstream description parameters of the bitstream do not include the second instruction identifier, the terminal device of the embodiment of FIG. 13 can determine whether the bitstream includes a third rendering parameter based on pre-agreed information. If the bitstream description parameters of the bitstream do not include the third instruction identifier, the terminal device of the embodiment of FIG. 13 can determine the type of the third rendering parameter in the analysis result based on pre-agreed information.

[0435] 14 is a schematic block diagram of an apparatus 1400 according to an embodiment of the present application. The apparatus 1400 may include a processor 1401 and a transceiver / transceiver pins 1402, and optionally further includes a memory device 1403.

[0436] The components of the device 1400 are coupled to one another through a bus 1404. In addition to a data bus, the bus 1404 further includes a power bus, a control bus, and a status signal bus. However, for clarity of explanation, the various types of buses in the figures will be referred to as the bus 1404.

[0437] Optionally, memory device 1403 may be configured to store instructions for the aforementioned method embodiments, and processor 1401 may be configured to execute the instructions in memory device 1403 and control the receive pin to receive a signal and the transmit pin to transmit a signal.

[0438] The apparatus 1400 may be an electronic device or a chip of an electronic device in the above-described method embodiments.

[0439] Any relevant content of the steps of the above method embodiments may be cited in the functional description of the corresponding functional modules, and will not be described in detail again here.

[0440] One embodiment further provides a computer-readable storage medium that stores computer instructions that, when executed on an electronic device, enable the electronic device to perform the steps of the associated methods described above to implement the encoding and decoding methods of the aforementioned embodiments.

[0441] An embodiment further provides a computer program product, which, when executed on a computer, enables the computer to perform the relevant steps described above to implement the encoding and decoding methods of the above embodiments.

[0442] In addition, an embodiment of the present application further provides an apparatus, which may specifically be a chip, a component, or a module, and may include a processor and a storage device connected thereto. The storage device is configured to store computer-executable instructions. When the apparatus is running, the processor can execute the computer-executable instructions stored in the storage device, so that the chip performs the encoding and decoding method of the above-mentioned method embodiment.

[0443] The electronic device, computer-readable storage medium, computer program product, or chip provided in the embodiments is configured to execute the corresponding method provided above. Therefore, for the beneficial effects that can be achieved, please refer to the beneficial effects of the corresponding method provided above. Details will not be described here.

[0444] Based on the above implementation description, those skilled in the art can understand that the above division into functional modules is used as an example for easy explanation. In actual application, the above functions may be assigned to different functional modules and implemented based on requirements. In other words, the internal structure of the device is divided into different functional modules to implement all or part of the above-described functions.

[0445] It should be understood that in some embodiments provided in the present application, the disclosed devices and methods may be implemented in other manners. For example, the described device embodiments are merely examples. For example, the division into modules or units is merely a division of logical functions, and actual implementation may involve other divisions. For example, multiple units or components may be combined or integrated into another device, or some features may be omitted or not implemented. In addition, the shown or discussed mutual couplings or direct couplings or communication connections may be implemented using some interfaces. Indirect couplings or communication connections between devices or units may be implemented in electronic, mechanical, or other forms.

[0446] The units described as separate parts may or may not be physically separate, and the parts shown as units may be one or more physical units, located in one place or distributed in different locations. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0447] In addition, the functional units of the embodiments of the present application may be integrated into one processing unit, each of the units may exist physically alone, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0448] All the contents of the embodiments of this application and all the contents of the same embodiments can be freely combined, and any combination of the aforementioned contents is within the scope of this application.

[0449] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application may essentially be implemented in the form of a software product, or a portion of the technical solutions may be implemented in the form of a software product. The software product may be stored in a storage medium and include instructions that instruct a device (such as a single-chip microcomputer or chip) or a processor to perform all or part of the steps of the methods described in the embodiments of the present application. The storage medium may include various media capable of storing program code, such as a USB flash drive, a removable hard disk drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0450] The above describes the embodiments of the present application with reference to the accompanying drawings. However, the present application is not limited to the specific implementations described above. The specific implementations described above are merely examples, not limitations. Those skilled in the art can make further modifications inspired by the present application without departing from the purpose of the present application and the scope of protection of the claims, and all such modifications shall fall within the scope of protection of the present application.

[0451] The method or algorithm steps described in connection with the contents disclosed in this embodiment of the present application may be implemented by hardware or by a processor executing software instructions. The software instructions may include corresponding software modules. The software modules may be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable hard disk, a compact disk read-only memory (CD-ROM), or any other form of storage medium known in the art. For example, the storage medium may be coupled to the processor such that the processor can read information from the storage medium and encode information into the storage medium. Of course, the storage medium may be a component of the processor. The processor and the storage medium may be located in an ASIC.

[0452] Those skilled in the art should recognize that in one or more of the above examples, the functions described in the embodiments of the present application may be implemented by hardware, software, firmware, or any combination thereof. If the functions are implemented by software, the functions may be stored on a computer-readable medium or transmitted as one or more instructions or code in a computer-readable medium. Computer-readable media include computer-readable storage media and communication media. Communication media include any medium that enables a computer program to be transmitted from one place to another. Storage media may be any available medium accessible to a general-purpose or special-purpose computer.

[0453] The above describes the embodiments of the present application with reference to the accompanying drawings. However, the present application is not limited to the specific implementations described above. The specific implementations described above are merely examples, not limitations. Those skilled in the art can make further modifications inspired by the present application without departing from the purpose of the present application and the scope of protection of the claims, and all such modifications shall fall within the scope of protection of the present application. [Explanation of symbols]

[0454] 1400 equipment 1401 processor 1402 Transceiver / Transceiver Pin 1403 Storage device 1404 Bus

Claims

1. A device-cloud collaboration system, comprising: a server and a terminal device, wherein the server comprises a first rendering module, an encoder, and a first communication module, and the terminal device comprises a second communication module, a second rendering module, and a decoder; the first rendering module is configured to perform a rendering process on a three-dimensional scene based on rendering parameters to obtain a rendered image, the rendering parameters including first rendering parameters obtained from the terminal device; select a first intermediate rendering result based on intermediate rendering results generated in the rendering process; and generate a virtual reference frame based on the first intermediate rendering result; the encoder is configured to predict the rendered image based on the virtual reference frame to obtain a predicted image, encode a residual image between the predicted image and the rendered image, and encode encoded data of the residual image into a bitstream, wherein the bitstream does not include encoded data of the first intermediate rendering result; the first communication module is configured to transmit the bitstream; the second communication module is configured to receive the bitstream; the decoder is configured to parse the bitstream to obtain an analysis result, the analysis result comprising a residual image corresponding to a current frame; the second rendering module is configured to perform a rendering process on the three-dimensional scene based on rendering parameters corresponding to the current frame, and generate a first intermediate rendering result in the rendering process, wherein the rendering parameters corresponding to the current frame include first rendering parameters generated by the terminal device, and generate a virtual reference frame based on the first intermediate rendering result generated by the second rendering module; the decoder is further configured to predict the current frame based on the virtual reference frame generated by the second rendering module to obtain a predicted image, and to perform reconstruction based on the predicted image determined by the decoder and the residual image corresponding to the current frame to obtain a reconstructed image of the current frame. Device-cloud collaborative system.

2. 1. An encoding method applied to a server, said method comprising: performing a rendering process on the three-dimensional scene based on rendering parameters to obtain a rendered image, the rendering parameters including first rendering parameters obtained from a terminal device; selecting a first intermediate rendering result based on the intermediate rendering results generated in the rendering process; generating a virtual reference frame based on the first intermediate rendering result; predicting the rendered image based on the virtual reference frame to obtain a predicted image; encoding a residual image between the predicted image and the rendered image, and encoding encoded data of the residual image into a bitstream, wherein the bitstream does not include encoded data of the first intermediate rendering result; A method comprising:

3. the bitstream further includes a first designation identifier and / or a second designation identifier; the first indication identifier indicates whether the bitstream includes the encoded data of the first intermediate rendering result; The method of claim 2 , wherein the second designation identifier indicates a type of the first intermediate rendering result.

4. The rendering parameters further include second rendering parameters generated by the server, and the method further comprises: encoding third rendering parameters into the bitstream, the third rendering parameters including all or part of the parameters of the second rendering parameters; 4. The method of claim 2 or 3, further comprising:

5. the rendering parameters further include the second rendering parameters generated by the server, and the bitstream further includes a third instruction identifier and / or a fourth instruction identifier; the third indication identifier indicates whether the bitstream includes the third rendering parameters, the third rendering parameters including all or part of the parameters of the second rendering parameters; The method of claim 2 , wherein the fourth indication identifier indicates a type of the third rendering parameter.

6. generating the virtual reference frame based on the first intermediate rendering result, determining the type of the first intermediate rendering result; generating the virtual reference frame based on the first intermediate rendering result and the type of the first intermediate rendering result; 6. The method of any one of claims 2 to 5, comprising:

7. generating the virtual reference frame based on the first intermediate rendering result and the type of the first intermediate rendering result, generating the virtual reference frame based on the first intermediate rendering result and a reconstructed image when the type of the first intermediate rendering result is a motion vector type; Including, the first intermediate rendering result is a computer graphics motion vector (CGMV), the CGMV being used to describe a displacement relationship between samples in the rendered image and samples in the reconstructed image; The method of claim 6.

8. generating the virtual reference frame based on the first intermediate rendering result and the type of the first intermediate rendering result, determining the intermediate rendering result as the virtual reference frame when the type of the first intermediate rendering result is determined to be an image type; Including, the first intermediate rendering result is an intermediate rendered image, and a computational complexity corresponding to the intermediate rendered image is lower than a computational complexity corresponding to the rendered image; The method of claim 6.

9. the first intermediate rendering result includes a CGMV and / or an intermediate rendered image, and a computational complexity corresponding to the intermediate rendered image is lower than a computational complexity corresponding to the rendered image; 9. The method according to any one of claims 2 to 8.

10. 1. A decoding method applied to a terminal device, said method comprising: receiving a bitstream; parsing the bitstream to obtain an analysis result, the analysis result including a residual image corresponding to a current frame; performing a rendering process on a three-dimensional scene based on rendering parameters corresponding to the current frame, and generating a first intermediate rendering result in the rendering process, wherein the rendering parameters include first rendering parameters generated by the terminal device; generating a virtual reference frame based on the first intermediate rendering result; predicting the current frame based on the virtual reference frame to obtain a predicted image; performing reconstruction based on the predicted image and the residual image to obtain a reconstructed image of the current frame; A method comprising:

11. the analysis result further includes a first instruction identifier and a second instruction identifier, the first instruction identifier indicating whether the bitstream includes encoded data of a first intermediate rendering result generated by a server, and the second instruction identifier indicating a type of the first intermediate rendering result generated by the server; The step of generating the first intermediate rendering result in the rendering process includes: generating the first intermediate rendering result in the rendering process based on the second instruction identifier when it is determined based on the first instruction identifier that the bitstream does not include the encoded data of the first intermediate rendering result generated by the server; 11. The method of claim 10, comprising:

12. the analysis result further includes a third instruction identifier and a fourth instruction identifier, the third instruction identifier indicating whether the bitstream includes a third rendering parameter, and the fourth instruction identifier indicating a type of the third rendering parameter; The method comprises: if it is determined based on the third instruction identifier that the bitstream includes the third rendering parameter, determining the type of the third rendering parameter based on the fourth instruction identifier; generating a fourth rendering parameter of another type based on the type of the third rendering parameter, the another type being one type other than the type of the third rendering parameter among a plurality of types corresponding to second rendering parameters generated by the server; further comprising the rendering parameters further include the third rendering parameter and the fourth rendering parameter; 12. The method of claim 10 or 11.

13. The method comprises: generating, by the terminal device, second rendering parameters corresponding to the current frame when it is determined based on the third indication identifier that the bitstream does not include the third rendering parameters; further comprising The rendering parameters further include second rendering parameters generated by the terminal device; The method of claim 12.

14. generating the virtual reference frame based on the first intermediate rendering result, determining the type of the first intermediate rendering result; generating the virtual reference frame based on the first intermediate rendering result and the type of the first intermediate rendering result; 14. The method of any one of claims 10 to 13, comprising:

15. generating the virtual reference frame based on the first intermediate rendering result and type information of the first intermediate rendering result, generating the virtual reference frame based on the first intermediate rendering result and a reconstructed image of another frame when the type of the first intermediate rendering result is a motion vector type; Including, the first intermediate rendering result is a computer graphics motion vector CGMV, the CGMV being used to describe a displacement relationship between samples in the reconstructed image of the current frame and samples in the reconstructed image of the other frame; The method of claim 14.

16. generating the virtual reference frame based on the first intermediate rendering result and type information of the first intermediate rendering result, determining the first intermediate rendering result as the virtual reference frame when the type of the first intermediate rendering result is determined to be an image type; Including, the first intermediate rendering result is an intermediate rendered image, and a computational complexity corresponding to the intermediate rendered image is lower than a computational complexity corresponding to the rendered image; The method of claim 14.

17. 17. The method of claim 10, wherein the first intermediate rendering result comprises a CGMV and / or an intermediate rendered image, and wherein a computational complexity corresponding to the intermediate rendered image is lower than a computational complexity corresponding to the rendered image.

18. A device-cloud collaboration system, comprising: a server and a terminal device, wherein the server comprises a first rendering module, an encoder, and a first communication module, and the terminal device comprises a second communication module, a second rendering module, and a decoder; the first rendering module is configured to perform a rendering process on a three-dimensional scene based on rendering parameters to obtain a rendered image, the rendering parameters including first rendering parameters obtained from the terminal device; select a first intermediate rendering result based on intermediate rendering results generated in the rendering process; and generate a virtual reference frame based on the first intermediate rendering result; the encoder is configured to predict the rendered image based on the virtual reference frame to obtain a predicted image, encode a residual image between the predicted image and the rendered image, encode encoded data of the residual image into a bitstream, encode a second intermediate rendering result, and encode encoded data of the second intermediate rendering result into the bitstream, wherein the second intermediate rendering result is a part of the first intermediate rendering result; the first communication module is configured to transmit the bitstream; the second communication module is configured to receive the bitstream; the decoder is configured to parse the bitstream to obtain an analysis result, the analysis result including a residual image corresponding to a current frame and a second intermediate rendering result corresponding to the current frame; the second rendering module is configured to perform a rendering process on the three-dimensional scene based on rendering parameters corresponding to the current frame, generate a third intermediate rendering result based on the second intermediate rendering result corresponding to the current frame in the rendering process, the rendering parameters corresponding to the current frame include first rendering parameters generated by the terminal device, generate a virtual reference frame based on the first intermediate rendering result, and the first intermediate rendering result used by the second rendering module to generate the virtual reference frame includes the second intermediate rendering result and the third intermediate rendering result corresponding to the current frame; the decoder is further configured to predict the current frame based on the virtual reference frame generated by the second rendering module to obtain a predicted image, and to perform reconstruction based on the predicted image determined by the decoder and the residual image corresponding to the current frame to obtain a reconstructed image of the current frame. Device-cloud collaborative system.

19. 1. An encoding method applied to a server, said method comprising: performing a rendering process on the three-dimensional scene based on rendering parameters to obtain a rendered image, the rendering parameters including first rendering parameters obtained from a terminal device; selecting a first intermediate rendering result based on the intermediate rendering results generated in the rendering process; generating a virtual reference frame based on the first intermediate rendering result; predicting the rendered image based on the virtual reference frame to obtain a predicted image; encoding a residual image between the predicted image and the rendered image, and encoding the encoded data of the residual image into a bitstream; encoding a second intermediate rendering result and encoding encoded data of the second intermediate rendering result into the bitstream, wherein the second intermediate rendering result is a part of the first intermediate rendering result; A method comprising:

20. The method of claim 19 , wherein the bitstream further includes a first instruction identifier, the first instruction identifier indicating a type and subtype of the second intermediate rendering result.

21. The rendering parameters further include second rendering parameters generated by the server, and the method further comprises: encoding third rendering parameters into the bitstream, the third rendering parameters including all or part of the parameters of the second rendering parameters; 21. The method of claim 19 or 20, further comprising:

22. the rendering parameters further include the second rendering parameters generated by the server, and the bitstream further includes a second instruction identifier and / or a third instruction identifier; the second indication identifier indicates whether the bitstream includes the third rendering parameters, and the third rendering parameters include all or part of the parameters of the second rendering parameters; 22. The method of claim 19, wherein the third indication identifier indicates a type of the third rendering parameter.

23. generating the virtual reference frame based on the first intermediate rendering result, determining a type of the first intermediate rendering result; generating the virtual reference frame based on the first intermediate rendering result and the type of the first intermediate rendering result; 23. The method of any one of claims 19 to 22, comprising:

24. generating the virtual reference frame based on the first intermediate rendering result and the type of the first intermediate rendering result, generating the virtual reference frame based on the first intermediate rendering result and a reconstructed image when the type of the first intermediate rendering result is a motion vector type; Including, the first intermediate rendering result is a computer graphics motion vector (CGMV), the CGMV being used to describe a displacement relationship between samples in the rendered image and samples in the reconstructed image; 24. The method of claim 23.

25. generating the virtual reference frame based on the first intermediate rendering result and the type of the first intermediate rendering result, determining the intermediate rendering result as the virtual reference frame when the type of the first intermediate rendering result is determined to be an image type; Including, the first intermediate rendering result is an intermediate rendered image, and a computational complexity corresponding to the intermediate rendered image is lower than a computational complexity corresponding to the rendered image; 24. The method of claim 23.

26. the first intermediate rendering result includes a CGMV and / or an intermediate rendered image, and a computational complexity corresponding to the intermediate rendered image is lower than a computational complexity corresponding to the rendered image; 24. The method of any one of claims 19 to 23.

27. 1. A decoding method applied to a terminal device, said method comprising: receiving a bitstream; parsing the bitstream to obtain an analysis result, the analysis result including a residual image corresponding to a current frame and a second intermediate rendering result corresponding to the current frame; performing a rendering process on a three-dimensional scene based on rendering parameters corresponding to the current frame, and generating a third intermediate rendering result based on the second intermediate rendering result in the rendering process, wherein the rendering parameters include first rendering parameters generated by the terminal device; generating a virtual reference frame based on a first intermediate rendering result, the first intermediate rendering result including the second intermediate rendering result and the third intermediate rendering result; predicting the current frame based on the virtual reference frame to determine a predicted image; performing reconstruction based on the predicted image and the residual image to obtain a reconstructed image of the current frame; A method comprising:

28. the decoded result further includes a first designation identifier, the first designation identifier indicating a type and a subtype of the second intermediate rendering result; The step of generating the third intermediate rendering result based on the second intermediate rendering result in the rendering processing step includes: determining the type and the subtype of the second intermediate rendering result based on the first designation identifier; generating a third intermediate rendering result of another subtype based on the type and subtype of the second intermediate rendering result in the rendering processing step, wherein the another subtype is one of subtypes included in the type corresponding to the second intermediate rendering result other than the subtype of the second intermediate rendering result; 28. The method of claim 27, comprising:

29. the analysis result further includes a second instruction identifier and a third instruction identifier, the second instruction identifier indicating whether the bitstream includes a third rendering parameter, and the third identifier indicating a type of the third rendering parameter; The method comprises: if it is determined based on the second instruction identifier that the bitstream includes the third rendering parameter, determining the type of the third rendering parameter based on the third instruction identifier; generating a fourth rendering parameter of another type based on the type of the third rendering parameter, the another type being one type other than the type of the third rendering parameter among a plurality of types corresponding to second rendering parameters generated by the server; further comprising the rendering parameters further include the third rendering parameter and the fourth rendering parameter; 29. The method of claim 27 or 28.

30. The method comprises: generating, by the terminal device, second rendering parameters corresponding to the current frame when it is determined based on the second indication identifier that the bitstream does not include the third rendering parameters; further comprising The rendering parameters further include second rendering parameters generated by the terminal device; 30. The method of claim 29.

31. generating the virtual reference frame based on the first intermediate rendering result, determining a type of the first intermediate rendering result; generating the virtual reference frame based on the first intermediate rendering result and the type of the first intermediate rendering result; 31. The method of any one of claims 27 to 30, comprising:

32. generating the virtual reference frame based on the first intermediate rendering result and type information of the first intermediate rendering result, generating the virtual reference frame based on the first intermediate rendering result and a reconstructed image of another frame when the type of the first intermediate rendering result is a motion vector; Including, the first intermediate rendering result is a computer graphics motion vector CGMV, the CGMV being used to describe a displacement relationship between samples in the reconstructed image of the current frame and samples in the reconstructed image of the other frame; 32. The method of claim 31.

33. generating the virtual reference frame based on the first intermediate rendering result and type information of the first intermediate rendering result, determining the first intermediate rendering result as the virtual reference frame when the type of the first intermediate rendering result is determined to be an image type; Including, the first intermediate rendering result is an intermediate rendered image, and a computational complexity corresponding to the intermediate rendered image is lower than a computational complexity corresponding to the rendered image; 32. The method of claim 31.

34. the first intermediate rendering result includes a CGMV and / or an intermediate rendered image, and a computational complexity corresponding to the intermediate rendered image is lower than a computational complexity corresponding to the rendered image; 34. The method of any one of claims 27 to 33.

35. A bitstream generation method, wherein a bitstream is generated based on the first rendering parameters according to any one of claims 10 to 17.

36. A bitstream generation method, wherein a bitstream is generated based on the first rendering parameters of any one of claims 27 to 34.

37. 1. An electronic device comprising: a memory and a processor, the memory coupled to the processor; The memory stores program instructions that, when executed by the processor, enable the electronic device to perform the method of any one of claims 2 to 17 or the method of any one of claims 19 to 34. Electronic devices.

38. 35. A chip comprising one or more interface circuits and one or more processors, the interface circuits configured to receive signals from a memory device of an electronic device and to transmit the signals to the processor, the signals comprising computer instructions stored in the memory device, the execution of the computer instructions by the processor enabling the electronic device to perform a method according to any one of claims 2 to 17 or any one of claims 19 to 34.

39. 35. A computer-readable storage medium storing a computer program that, when executed on a computer or a processor, enables the computer or the processor to perform the method of any one of claims 2 to 17 or any one of claims 19 to 34.

40. 35. A computer program product comprising computational instructions which, when executed by a computer or processor, cause the steps of the method of any one of claims 2 to 17 to be performed or the steps of the method of any one of claims 19 to 34 to be performed.

41. 1. An apparatus for storing a bitstream, said apparatus comprising: a receiver and at least one storage medium; the receiver is configured to receive the bitstream; the at least one storage medium is configured to store the bitstream; The bitstream is generated according to the encoding method of any one of claims 2 to 9, or according to the encoding method of any one of claims 19 to 26, or according to the bitstream generation method of claim 35 or claim 36. Device.

42. 1. An apparatus for transmitting a bitstream, said apparatus comprising: a transmitter and at least one storage medium; the at least one storage medium is configured to store the bitstream, the bitstream being generated according to the encoding method of any one of claims 2 to 9, or according to the encoding method of any one of claims 19 to 26, or according to the bitstream generation method of claim 35 or claim 36; the transmitter is configured to obtain the bitstream from the storage medium and transmit the bitstream to a device-side device over a transmission medium; Device.

43. 1. A system for distributing a bitstream, the system comprising: at least one storage medium configured to store at least one bitstream, wherein said at least one bitstream is generated according to the encoding method of any one of claims 2 to 9, or according to the encoding method of any one of claims 19 to 26, or according to the bitstream generation method of claim 35 or claim 36; and a streaming media device configured to obtain a target bitstream from the at least one storage medium and transmit the target bitstream to a device-side device, the streaming media device comprising a content server or a content distribution server; A system comprising:

Images