Coding and decoding method and end-cloud cooperative system

Abstract

The embodiment of the invention provides a coding and decoding method and an end-cloud cooperation system. The coding and decoding method is realized based on the end-cloud collaborative system, and the total code rate of the end-cloud can be reduced under the condition of ensuring the end-side rendering picture. The encoding method comprises the steps that rendering processing is carried out on a three-dimensional scene to obtain a rendered image, a rendering intermediate result is generated in the rendering processing process, and the rendering intermediate result comprises a first rendering intermediate result. And performing target operation on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block. And predicting the rendered image based on the virtual reference block to obtain a predicted image. And encoding a residual image between the predicted image and the rendered image to obtain a code stream. Wherein the target operation comprises at least one of reprojection, validity check, weighted averaging, hole filling or intra-frame noise reduction.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of coding and decoding, and in particular to a coding and decoding method and a terminal-cloud collaborative system. Background Art

[0002] Many scenarios (such as games, virtual reality (VR) / augmented reality (AR), etc.) require rendering to generate images to make the images more realistic and improve the user experience. Among them, rendering requires relatively strong computing power, and the computing power of the end-side device is often far weaker than that of the cloud-side server due to objective physical conditions such as device size and power consumption; therefore, rendering is usually deployed on the cloud-side server, which performs rendering and then compresses the rendered image / video and sends it to the end-side device for display.

[0003] As people's requirements for rendering quality continue to increase and the clarity of display devices continues to improve, the image quality and resolution of rendered images / videos are also continuously improved, which will lead to an increase in the bit rate overhead of the rendered images / videos after compression.

[0004] In the prior art, the cloud-side server usually encodes and transmits the rendered low-resolution image / video, and transmits the rendering intermediate results generated in the process of rendering the high-resolution image / video to the end-side device. The end-side device uses the rendering intermediate results sent by the cloud-side server to upsample the low-resolution image / video sent by the cloud-side rendering, generate a high-resolution image / video to be displayed, and display it. In this way, although the bit rate overhead can be reduced to a certain extent, the encoding efficiency is still relatively low. Summary of the invention

[0005] In view of this, the present application provides a coding method and a terminal-cloud collaborative system. The coding method is implemented based on the terminal-cloud collaborative system, and can reduce the total bit rate of the terminal-cloud while ensuring the terminal-side rendering of the picture.

[0006] In a first aspect, an embodiment of the present application provides a coding method, which is applied to a cloud-side device, and the method includes: rendering a three-dimensional scene to obtain a rendered image, and the rendering intermediate result is generated during the above rendering process, and the above rendering intermediate result includes a first rendering intermediate result. Based on the above first rendering intermediate result, a target operation is performed on the reconstructed image to generate a virtual reference block. Based on the above virtual reference block, the above rendered image is predicted to obtain a predicted image. The residual image between the above predicted image and the above rendered image is encoded to obtain a code stream. Among them, the above target operation includes at least one of reprojection, validity check, weighted average, hole filling or intra-frame noise reduction.

[0007] The method provided in the embodiment of the present application can generate a virtual reference block based on the intermediate rendering result generated in the rendering process by means of reprojection, validity check, weighted average, hole filling, intra-frame noise reduction, etc. The generated virtual reference block can provide a more accurate reference for encoding and decoding of the rendered picture, thereby improving the accuracy of inter-frame prediction, thereby improving the encoding efficiency and reducing the bit rate. In this way, it is possible to reduce the total bit rate of the end cloud while ensuring the end-side rendering of the picture.

[0008] It can be understood that the virtual reference frame is also a virtual reference block, that is, the virtual reference frame can be regarded as a virtual reference block.

[0009] In a possible implementation manner, when a target condition is met, a target operation is performed on a reconstructed image of the encoded image based on the first rendering intermediate result.

[0010] For example, the above target conditions may be related to the terminal device capability, target similarity, or prediction module parameters.

[0011] The target similarity is the similarity between the virtual reference block and the frame to be encoded, and the prediction module parameter is used to indicate whether to use inter-frame prediction and / or whether the virtual reference block is used in inter-frame prediction.

[0012] For example, the above-mentioned goal may be satisfied by using inter-frame prediction.

[0013] For another example, the above-mentioned target may be satisfied by using a virtual reference block for inter-frame prediction.

[0014] For another example, the above-mentioned target satisfaction may be that the terminal side device capability satisfies the virtual reference frame generation requirement.

[0015] The method provided in the embodiment of the present application determines whether to execute the target operation through the target condition, thereby ensuring that the total bit rate of the end cloud is reduced while the end-side rendering screen is not affected.

[0016] In a possible implementation, the code stream includes first indication information, and when the first indication information is a first value, the code stream further includes second indication information; and / or when the second indication information is a second value, the code stream further includes third indication information.

[0017] For example, the first indication information may indicate whether the target operation is executed. The second indication information may indicate the method adopted by the target operation. The second indication information may indicate the parameters involved in the method adopted by the target operation.

[0018] For another example, the first indication information being a first value may indicate that the target step is executed, and the second indication information being a second value may indicate the method used for the target operation.

[0019] For another example, if the target operations include reprojection, validity check, weighted averaging, hole filling and intra-frame noise reduction, the first indication information may be 5 flag values, and the 5 flag values ​​are respectively used to indicate whether reprojection, validity check, weighted averaging, hole filling and intra-frame noise reduction are performed.

[0020] The method provided in the embodiment of the present application can also adjust the generation steps and the content of the generation steps of the virtual reference block according to the terminal device capabilities, thereby ensuring that the terminal rendering picture is not affected while the total bit rate of the terminal cloud is reduced.

[0021] In a possible implementation manner, it may also be determined whether to generate a virtual reference block based on device capability information.

[0022] For example, when the cloud-side device knows that the terminal-side device is capable of generating all virtual reference blocks, the encoder can autonomously choose whether to encode using the original reference blocks or by generating virtual reference blocks.

[0023] For another example, when the cloud side knows that the terminal side device is not capable of generating all virtual reference blocks, all virtual reference blocks are generated only on the cloud side device, and the encoder is allowed to autonomously select virtual reference blocks or original reference blocks for encoding.

[0024] In a possible implementation manner, the code stream includes fourth indication information. When the fourth indication information is a fourth value, the code stream further includes fifth indication information.

[0025] In a possible implementation manner, the fourth indication information and / or the fifth indication information may be determined according to device capability information.

[0026] In a possible implementation manner, the fourth indication information is used to indicate the number of generated virtual reference blocks.

[0027] In a possible implementation, the fifth indication information is used to indicate the virtual reference block selection result, that is, to indicate which virtual reference blocks are generated.

[0028] In a possible implementation manner, the first rendering intermediate result may be determined according to the target operation.

[0029] The method provided in the embodiment of the present application can select the required first rendering intermediate result from the rendering intermediate results through the target operation, thereby reducing the number of rendering intermediate results and improving rendering efficiency.

[0030] In a possible implementation, the reprojected pixel points in the reconstructed image can be validated based on the first rendering intermediate result to obtain a validity test result. The validity test result is used to indicate whether the pixel points to be detected of the virtual reference block to be generated have passed the validity test. The validity test result is used to generate the virtual reference block.

[0031] It is understandable that some pixels in the virtual reference block to be generated may be blocked by foreground objects in the reconstructed image, and are only exposed in the current frame when the virtual reference block is generated, so the corresponding pixels cannot be found in the reconstructed image. For these pixels, if they are not eliminated, wrong pixel values ​​will be obtained in the virtual reference block, forming "afterimages". For this reason, the embodiment of the present application determines the defective pixels by reprojecting the pixels of the pixels to be detected of the virtual reference block to be generated in the above-mentioned reconstructed image, and eliminates the defective pixels in the reprojection, so as to obtain a more accurately predicted virtual reference block and improve coding efficiency.

[0032] In a possible implementation, the reprojected pixel point is inversely reprojected into the virtual reference block to be generated based on the first rendering intermediate result to obtain the inversely reprojected pixel coordinates of the reprojected pixel point; and the validity check result is obtained based on the inversely reprojected pixel coordinates and the pixel coordinates of the pixel point to be detected.

[0033] It can be seen that the method provided in the embodiment of the present application can determine the defective pixels through the pixel coordinates of the pixel to be detected, eliminate the defective pixels in the reprojection, thereby obtaining a more accurately predicted virtual reference block and improving the coding efficiency.

[0034] In a possible implementation method, the above-mentioned reprojected pixel points and the above-mentioned pixel points to be detected can be projected into the world coordinate system based on the above-mentioned first rendering intermediate result to obtain the world coordinates of the above-mentioned reprojected pixel points and the world coordinates of the above-mentioned pixel points to be detected; based on the world coordinates of the above-mentioned reprojected pixel points and the world coordinates of the above-mentioned pixel points to be detected, the above-mentioned validity check result can be obtained.

[0035] It can be seen that the method provided in the embodiment of the present application can determine the defective pixels through the world coordinates of the pixel points to be detected, eliminate the defective pixels in the reprojection, thereby obtaining a more accurately predicted virtual reference block and improving the coding efficiency.

[0036] In a possible implementation, the validity check result can be obtained based on the first rendering intermediate result, the target information corresponding to the reprojected image point and the target information of the pixel to be detected, wherein the target information includes at least one of an object ID, a mesh ID or a material ID.

[0037] In a possible implementation, the reconstructed image may be weighted averaged based on the first rendering intermediate result to obtain a weighted average result, wherein the weighted average result is used to indicate the weighted average pixel value of the virtual reference block to be generated, and the weighted average result is used to generate the virtual reference block.

[0038] It can be seen that in this embodiment, more reasonable pixel values ​​can be provided for the pixel points in the virtual reference block through the pixel values ​​of the pixel points in the reconstructed image.

[0039] In a possible implementation, a weighted average is performed based on the first rendering intermediate result, the pixel values ​​of the pixels in the reconstructed image, and the image sequence number of the reconstructed image to obtain the weighted average result.

[0040] It can be understood that the closer the image sequence number is to the virtual reference block, the higher the correlation between the reconstructed image and the virtual reference block, so higher weights can be assigned to closer reprojected pixel values ​​and smaller weights can be assigned to farther reprojected pixel values, thereby providing more reasonable pixel values ​​for the pixels in the virtual reference block.

[0041] In a possible implementation, a weighted average may be performed based on the first rendering intermediate result, the pixel value of the pixel in the reconstructed image, the world coordinates of the pixel in the reconstructed image, and the world coordinates of the pixel of the virtual reference block to be generated to obtain the weighted average result.

[0042] It can be understood that the smaller the pixel world coordinate error between the reconstructed image and the virtual reference block to be generated, the higher the correlation between the reconstructed image and the virtual reference block to be generated, so a higher weight can be assigned to the reprojected pixel value with a small pixel world coordinate error; the larger the pixel world coordinate error between the reconstructed image and the virtual reference block to be generated, the smaller the correlation between the reconstructed image and the virtual reference block to be generated, so a lower weight can be assigned to the reprojected pixel value with a small pixel world coordinate error, thereby providing a more reasonable pixel value for the pixel points in the virtual reference block.

[0043] In a possible implementation, a weighted average is performed based on the first rendering intermediate result, the pixel value of the pixel in the reconstructed image, the pixel coordinates of the pixel in the reconstructed image, and the pixel coordinates of the pixel of the virtual reference block to be generated to obtain the weighted average result.

[0044] It can be understood that the smaller the pixel coordinate error between the reconstructed image and the virtual reference block to be generated, the higher the correlation between the reconstructed image and the virtual reference block to be generated, so a higher weight can be assigned to the reprojected pixel value with a small pixel coordinate error, and the larger the pixel coordinate error between the reconstructed image and the virtual reference block to be generated, the smaller the correlation between the reconstructed image and the virtual reference block to be generated, so a lower weight can be assigned to the reprojected pixel value with a small pixel coordinate error, thereby providing a more reasonable pixel value for the pixel point in the virtual reference block.

[0045] In a possible implementation, holes in the reconstructed image can be filled based on the first rendering intermediate result to obtain a hole filling result, wherein the hole filling result is used to indicate the pixel value of the pixel to be filled after the hole is filled, and the pixel to be filled is the pixel that fails the validity check in the virtual reference block to be generated, and the hole filling result is used to generate the virtual reference block.

[0046] It can be seen that the method provided in the embodiment of the present application can fill holes in the pixel points that fail the validity check in the generated virtual reference block to replace the erroneous predicted pixel values ​​with more reasonable predicted pixel values, thereby improving the accuracy of the virtual reference block or the virtual reference block and improving the coding efficiency.

[0047] In a possible implementation, holes in the reconstructed image may be filled based on the first rendering intermediate result and the reprojected pixel points of the reconstructed image to obtain the hole filling result.

[0048] It can be seen that the method provided in the embodiment of the present application can fill holes in the pixel points that fail the validity check in the to-be-generated virtual reference block by reprojecting the pixel points in the reconstructed image, so as to replace the erroneous predicted pixel values ​​with more reasonable predicted pixel values, thereby improving the accuracy of the virtual reference block or the virtual reference block and improving the coding efficiency.

[0049] In a possible implementation, the reconstructed image can be hole-filled based on the first rendering intermediate result and the target pixel point of the reprojected pixel point of the reconstructed image to obtain a hole-filling result, wherein the target pixel point is a pixel point in the reconstructed image whose distance from the reprojected pixel point is less than a first threshold.

[0050] It can be seen that the method provided in the embodiment of the present application can fill holes in the pixel points that fail the validity check in the to-be-generated virtual reference block through the target pixel points of the reprojected pixel points in the reconstructed image, so as to replace the erroneous predicted pixel values ​​with more reasonable predicted pixel values, thereby improving the accuracy of the virtual reference block or the virtual reference block and improving the coding efficiency.

[0051] In a possible implementation, holes in the reconstructed image may be filled based on the first rendering intermediate result and non-to-be-filled pixels to obtain the hole filling result. The non-to-be-filled pixels are pixels that have passed the validity check in the virtual reference block to be generated.

[0052] It can be seen that the method provided in the embodiment of the present application can fill holes in the pixel points in the virtual reference block to be generated that have not passed the validity check by using the pixel points in the virtual reference block to be generated that have passed the validity check, so as to replace the erroneous predicted pixel values ​​with more reasonable predicted pixel values, thereby improving the accuracy of the virtual reference block or the virtual reference block and improving the coding efficiency.

[0053] In a second aspect, an embodiment of the present application also provides a decoding method, which is applied to an end-side device, and the method includes: receiving a code stream; decoding the above code stream to obtain a residual image of the current image; rendering a three-dimensional scene to obtain a rendered image, wherein the rendering intermediate result is generated during the above rendering process, and the above rendering intermediate result includes a first rendering intermediate result; based on the above first rendering intermediate result, a target operation is performed on the reconstructed image to generate a virtual reference block, and the above target operation includes at least one of reprojection, validity check, weighted averaging, hole filling or intra-frame noise reduction; based on the above virtual reference block, the rendered image is predicted to obtain a predicted image; based on the above predicted image and the above residual image, a reconstruction is performed to obtain a reconstructed image of the above current frame.

[0054] In a possible implementation, the terminal-side device information may be sent to the cloud-side device.

[0055] In a possible implementation, the above-mentioned code stream can be decoded to obtain a residual image of the current image; the above-mentioned code stream is decoded to obtain indication information, the above-mentioned indication information includes first indication information, and when the first indication information is a first value, the above-mentioned indication information also includes second indication information; and / or, when the second indication information is a second value, the above-mentioned indication information also includes third indication information.

[0056] In one possible implementation, indication information is received from a cloud-side device, where the indication information includes first indication information, and when the first indication information is a first value, the indication information also includes second indication information; and / or when the second indication information is a second value, the indication information also includes third indication information.

[0057] In a possible implementation manner, the first rendering intermediate result is determined according to the target operation.

[0058] In a possible implementation, the reprojected pixel points in the reconstructed image can be validated based on the first rendering intermediate result to obtain a validity test result. The validity test result is used to indicate whether the pixel points to be detected of the virtual reference block to be generated have passed the validity test. The validity test result is used to generate the virtual reference block.

[0059] In a possible implementation, the reprojected pixel point can be inversely reprojected into the virtual reference block to be generated based on the first rendering intermediate result to obtain the inversely reprojected pixel coordinates of the reprojected pixel point; and the validity check result can be obtained based on the inversely reprojected pixel coordinates and the pixel coordinates of the pixel point to be detected.

[0060] In a possible implementation method, the above-mentioned reprojected pixel points and the above-mentioned pixel points to be detected can be projected into the world coordinate system based on the above-mentioned first rendering intermediate result to obtain the world coordinates of the above-mentioned reprojected pixel points and the world coordinates of the above-mentioned pixel points to be detected; based on the world coordinates of the above-mentioned reprojected pixel points and the world coordinates of the above-mentioned pixel points to be detected, the above-mentioned validity check result can be obtained.

[0061] In a possible implementation, the validity check result can be obtained based on the first rendering intermediate result, the target information corresponding to the reprojected image point and the target information of the pixel to be detected, wherein the target information includes at least one of an object ID, a mesh ID or a material ID.

[0062] In a possible implementation, the reconstructed image may be weighted averaged based on the first rendering intermediate result to obtain a weighted average result, wherein the weighted average result is used to indicate the weighted average pixel value of the virtual reference block to be generated, and the weighted average result is used to generate the virtual reference block.

[0063] In a possible implementation, a weighted average may be performed based on the first rendering intermediate result, the pixel values ​​of the pixels in the reconstructed image, and the image sequence number of the reconstructed image to obtain the weighted average result.

[0064] In a possible implementation, a weighted average may be performed based on the first rendering intermediate result, the pixel value of the pixel in the reconstructed image, the world coordinates of the pixel in the reconstructed image, and the world coordinates of the pixel of the virtual reference block to be generated to obtain the weighted average result.

[0065] In a possible implementation, a weighted average may be performed based on the first rendering intermediate result, the pixel value of the pixel in the reconstructed image, the pixel coordinates of the pixel in the reconstructed image, and the pixel coordinates of the pixel of the virtual reference block to be generated to obtain the weighted average result.

[0066] In a possible implementation, holes in the reconstructed image can be filled based on the first rendering intermediate result to obtain a hole filling result, wherein the hole filling result is used to indicate the pixel value of the pixel to be filled after the hole is filled, and the pixel to be filled is the pixel that fails the validity check in the virtual reference block to be generated, and the hole filling result is used to generate the virtual reference block.

[0067] In a possible implementation, holes in the reconstructed image may be filled based on the first rendering intermediate result and the reprojected pixel points of the reconstructed image to obtain the hole filling result.

[0068] In a possible implementation, the reconstructed image can be hole-filled based on the first rendering intermediate result and the target pixel point of the reprojected pixel point of the reconstructed image to obtain a hole-filling result, wherein the target pixel point is a pixel point in the reconstructed image whose distance from the reprojected pixel point is less than a first threshold.

[0069] In a possible implementation, holes in the reconstructed image may be filled based on the first rendering intermediate result and non-to-be-filled pixels to obtain the hole filling result. The non-to-be-filled pixels are pixels that have passed the validity check in the virtual reference block to be generated.

[0070] In a third aspect, an embodiment of the present application also provides a coding device, including a rendering module, an encoder and a communication module. The rendering module is used to render a three-dimensional scene to obtain a rendered image, and the rendering intermediate result is generated during the rendering process, and the rendering intermediate result includes a first rendering intermediate result; the rendering module is also used to perform a target operation on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block, and the target operation includes at least one of reprojection, validity check, weighted average, hole filling or intra-frame noise reduction; the encoder is used to predict the rendered image based on the virtual reference block to obtain a predicted image; the encoder is also used to encode the residual image between the predicted image and the rendered image to obtain a code stream; the communication module is used to send the code stream.

[0071] In a fourth aspect, an embodiment of the present application also provides a decoding device, including a rendering module, a decoder and a communication module. The communication module is used to receive a code stream. The decoder is used to decode the code stream to obtain a residual image of the current image. The rendering module is used to render a three-dimensional scene to obtain a rendered image, wherein the rendering intermediate result is generated during the rendering process, and the rendering intermediate result includes a first rendering intermediate result. The rendering module is also used to perform a target operation on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block, and the target operation includes at least one of reprojection, validity check, weighted average, hole filling or intra-frame noise reduction. The decoder is also used to predict the rendered image based on the virtual reference block to obtain a predicted image. The decoder is also used to reconstruct based on the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0072] In a fifth aspect, an embodiment of the present application further provides a terminal-cloud collaborative system, comprising an encoding device in any one of the implementations of the third aspect and a decoding device in any one of the implementations of the fourth aspect.

[0073] In the sixth aspect, an embodiment of the present application provides a cloud-side device, including: a memory and a processor, the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the server executes the encoding method in the first aspect or any possible implementation of the first aspect.

[0074] In the seventh aspect, an embodiment of the present application provides an end-side device, including: a memory and a processor, the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the terminal device executes the decoding method in the second aspect or any possible implementation of the second aspect.

[0075] In the eighth aspect, an embodiment of the present application provides a chip, comprising one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of a cloud-side device and send a signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the cloud-side device executes the encoding method in the first aspect or any possible implementation of the first aspect.

[0076] In the ninth aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an end-side device and send a signal to the processor, the signal comprising a computer instruction stored in the memory; when the processor executes the computer instruction, the end-side device executes the decoding method in the second aspect or any possible implementation of the second aspect.

[0077] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect.

[0078] In the eleventh aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the decoding method in the second aspect or any possible implementation of the second aspect.

[0079] In the twelfth aspect, an embodiment of the present application provides a computer program product, which includes computing instructions. When the computing instructions are executed by a computer or a processor, the computer or the processor executes the encoding method in the first aspect or any possible implementation of the first aspect.

[0080] In the thirteenth aspect, an embodiment of the present application provides a computer program product, which includes computing instructions. When the computing instructions are executed by a computer or a processor, the computer or the processor executes the decoding method in the second aspect or any possible implementation of the second aspect.

[0081] In a fourteenth aspect, an embodiment of the present application provides an encoder, comprising a processing circuit for executing the encoding method according to the first aspect or any possible implementation method.

[0082] In a fifteenth aspect, an embodiment of the present application provides a decoder, comprising a processing circuit for executing a decoding method according to the second aspect or any possible implementation method.

[0083] In the sixteenth aspect, an embodiment of the present application provides an encoder, comprising: one or more processors; a non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein when the program is executed by the processor, the encoder performs the encoding method in the first aspect or any possible implementation method.

[0084] In the seventeenth aspect, an embodiment of the present application provides a decoder, comprising: one or more processors; a non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein when the program is executed by the processor, the decoder performs the decoding method in the second aspect or any possible implementation method.

[0085] In the eighteenth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, including program code, which, when executed by a computer device, is used to execute the method according to the first aspect, the second aspect or any possible implementation method.

[0086] In a nineteenth aspect, an embodiment of the present application provides a non-transitory storage medium, comprising a code stream according to the method in the first aspect, the second aspect or any possible implementation manner.

[0087] The computer storage medium, computer program product and chip provided in this embodiment are all used to execute the method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the method provided above and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0089] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0090] Figure 2 A schematic diagram of a terminal-cloud collaboration system provided in an embodiment of the present application;

[0091] Figure 3 A schematic diagram of an encoding process provided in an embodiment of the present application;

[0092] Figure 4 A schematic diagram of a validity verification process provided in an embodiment of the present application;

[0093] Figure 5 A schematic diagram of a coding framework provided in an embodiment of the present application;

[0094] Figure 6 Schematic diagram of another coding framework provided in an embodiment of the present application

[0095] Figure 7 A schematic diagram of the decoding process provided in an embodiment of the present application;

[0096] Figure 8 A schematic diagram of a decoding framework provided in an embodiment of the present application;

[0097] Fig. 9 Another schematic diagram of a decoding framework provided in an embodiment of the present application;

[0098] Fig.10 A schematic diagram of the structure of a chip provided in an embodiment of the present application;

[0099] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0100] Fig.12 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0101] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0102] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0103] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0104] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0105] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0106] Before introducing the solutions provided by the embodiments of the present application, the terms involved in the embodiments of the present application are first explained.

[0107] Rendering intermediate results: The intermediate data used to generate the image / video to be displayed generated by the graphics rendering engine during the process of generating the image / video to be displayed. Including computer graphic motion vector (CGMV) generated by computer graphics methods, low-quality rendering results, position map, normal map, albedo map, specular intensity map, Mesh ID, Material ID, depth map, camera information (such as the change of camera position in each frame), and other information generated during the rendering process.

[0108] Rendering parameters: input to the graphics rendering engine, including various parameters used for rendering, the position and color vectors of all light sources, the position vector of the player or observer, the sampling method of each texture, the position coordinates of each scene object, the motion trajectory of moving objects, and skeletal animation parameters.

[0109] Auxiliary information: including rendering intermediate results and rendering parameters.

[0110] Stream auxiliary information: write auxiliary information of the stream.

[0111] CGMV: Data generated by the graphics engine that describes the correspondence between the pixels of any two frames of rendered images.

[0112] Virtual reference frame: A virtual reference frame is also a form of virtual reference block. It is a reference frame based on prior information. The prior information includes the decoded information inside the encoder and information other than the video to be encoded. The virtual reference frame can be used as supplementary information for inter-frame coding to further remove video time domain redundancy.

[0113] Reprojection: refers to the calculation process of finding the corresponding pixel point in the historical frame (reconstructed image) for a pixel in the reconstructed frame based on one or more of the rendering intermediate result depth, camera information, forward CGMV and other information. The forward in the forward CGMV refers to the previous frame, that is, the image frame earlier in time or the image frame with an earlier image sequence number.

[0114] Inverse reprojection: refers to the calculation process of finding the corresponding pixel point in the reconstructed frame based on one or more of the rendering intermediate result depth, camera information, backward CGMV and other information for the reprojected pixel point in the historical frame of a certain pixel in the reconstructed frame. Among them, the backward in the backward CGMV refers to the later frame, that is, the image frame with a later time or an image frame with a later image sequence number.

[0115] Validity check: Due to the limitations of prior information, there are some incorrectly predicted pixels in the above reprojection results. The validity can be verified by using one or more of the rendering intermediate result depth and camera information, backward CGMV and other information.

[0116] Afterimage: In the process of generating a virtual reference frame, due to the lack of the above-mentioned prior information, the predicted pixel values ​​found using the CGMV information are inaccurate, resulting in image defects in the virtual reference frame.

[0117] Hole: A collection of pixels without content in the virtual reference frame, including points where the predicted pixel value cannot be found due to insufficient prior information, and empty pixels formed after the afterimage elimination operation.

[0118] Filtering: refers to the operation of eliminating noise in the virtual reference frame.

[0119] The embodiment of the present application provides a coding method, which is implemented based on the end-cloud collaborative system and can reduce the total bit rate of the end-cloud while ensuring the end-side rendering of the picture. The cloud-side device can be a server, a terminal device (such as a mobile phone) or other devices. The end-side device can be a terminal device or other devices.

[0120] The following takes the cloud-side device as a server and the end-side device as a terminal device as an example to introduce the encoding and decoding method provided in the embodiment of the present application.

[0121] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application. Figure 1 The application scenario shown is a terminal-cloud collaborative rendering scenario. This application can be applied to various terminal-cloud collaborative scenarios; for example, cloud games, VR / AR conferences, etc., which are not limited in this application. This application is explained using cloud games as an example.

[0122] Reference Figure 1In one possible way, a terminal device such as a mobile phone, a personal computer (PC), VR glasses, an AR helmet, a tablet computer, and other terminal devices can send the first rendering parameter to a central server (such as a game server) through an edge server. Then, the central server can render the three-dimensional scene based on the rendering parameter (the rendering parameter can only include the first rendering parameter uploaded by the terminal device, or include the first rendering parameter uploaded by the terminal device and the second rendering parameter generated by the central server) to obtain a rendered image (such as a game image). After that, the central server encodes the rendered image based on part of the rendering intermediate results (hereinafter referred to as the first rendering intermediate result) in the rendering intermediate result generated during the rendering process to obtain a code stream and distribute the code stream to each terminal device through the edge server. Subsequently, the terminal device can render the three-dimensional scene based on the rendering parameter (the rendering parameter includes the first rendering parameter and the fifth rendering parameter, the fifth rendering parameter is also generated by the terminal device, and the fifth rendering parameter can include the same parameter as the second rendering parameter), and generate the first rendering intermediate result during the rendering process; then the terminal device decodes the code stream based on the first rendering intermediate result to obtain a reconstructed image.

[0123] Reference Figure 1 In one possible way, the first rendering parameter can be sent to the central server by a terminal device such as a mobile phone, a PC, a VR glasses, an AR helmet, a tablet computer, and other terminal devices through an edge server. Then, the central server can render the three-dimensional scene based on the rendering parameter (the rendering parameter can only include the first rendering parameter uploaded by the terminal device, or include the first rendering parameter uploaded by the terminal device and the second rendering parameter generated by the server) to obtain a rendered image. After that, the central server encodes the rendered image based on the first rendering intermediate result generated during the rendering process; and encodes the second rendering intermediate result (the second rendering intermediate result is part of the first rendering intermediate result) to obtain a code stream and distribute the code stream to each terminal device through the edge server. Subsequently, the terminal device can render the three-dimensional scene based on the rendering parameter (the rendering parameter includes the first rendering parameter and the fifth rendering parameter), and generate a third rendering intermediate result (the third rendering intermediate result is the part of the first rendering intermediate result other than the second rendering intermediate result) during the rendering process; then, the terminal device generates a first rendering intermediate result based on the second rendering intermediate result and the third rendering intermediate result, and decodes the code stream based on the first rendering intermediate result to obtain a reconstructed image.

[0124] Reference Figure 1In one possible way, the first rendering parameters can be sent to the central server by a terminal device such as a mobile phone, PC, VR glasses, AR helmet, tablet computer, or other terminal devices through an edge server. Then, the central server can render the three-dimensional scene based on the rendering parameters (the rendering parameters include the first rendering parameters uploaded by the terminal device and the second rendering parameters generated by the central server) to obtain a rendered image. Afterwards, the central server encodes the rendered image based on the first rendering intermediate result generated during the rendering process to obtain a code stream and writes the third rendering parameters (the third rendering parameters include all or part of the parameters in the second rendering parameters) into the code stream; then, the central server distributes the code stream to each terminal device through the edge server. Subsequently, the terminal device may perform rendering processing on the three-dimensional scene based on the rendering parameters (when the third rendering parameters include all parameters of the second rendering parameters, the rendering parameters may include the first rendering parameters and the third rendering parameters parsed from the code stream; when the third rendering parameters include part of the parameters of the second rendering parameters, the rendering parameters may include the first rendering parameters, the third rendering parameters parsed from the code stream, and the fourth rendering parameters generated by the terminal device, and the fourth rendering parameters are the parameters of the second rendering parameters other than the third rendering parameters), and generate a first rendering intermediate result during the rendering process; then, the terminal device decodes the code stream based on the first rendering intermediate result to obtain a reconstructed image.

[0125] Reference Figure 1In one possible way, a terminal device such as a mobile phone, PC, VR glasses, AR helmet, tablet computer, or other terminal devices can send the first rendering parameter to the central server through the edge server. Then, the central server can render the three-dimensional scene based on the rendering parameters (including the first rendering parameters uploaded by the terminal device and the second rendering parameters generated by the central server) to obtain a rendered image. Afterwards, the central server encodes the rendered image based on the first rendering intermediate result generated during the rendering process, and encodes the second rendering intermediate result to obtain a code stream and write the third rendering parameter into the code stream; then, the central server distributes the code stream to each terminal device through the edge server. Subsequently, the terminal device can perform rendering processing on the three-dimensional scene based on the rendering parameters (when the third rendering parameters include all parameters of the second rendering parameters, the rendering parameters may include the first rendering parameters and the third rendering parameters parsed from the code stream; when the third rendering parameters include part of the parameters of the second rendering parameters, the rendering parameters may include the first rendering parameters, the third rendering parameters parsed from the code stream, and the fourth rendering parameters generated by the terminal device), and generate a third rendering intermediate result during the rendering process; then the terminal device generates a first rendering intermediate result based on the second rendering intermediate result and the third rendering intermediate result, and decodes the code stream based on the first rendering intermediate result to obtain a reconstructed image.

[0126] In this way, part or all of the rendering is executed on the terminal device, and the server may not send the rendering intermediate result or only send part of the rendering intermediate result to the terminal device; therefore, the present application can reduce the interaction delay while ensuring that the bit rate overhead of the data stream transmitted from the server to the terminal device is effectively reduced. And because the rendering intermediate result is highly correlated with the rendered image, the present application encodes the rendered image based on the rendering intermediate result to ensure the quality of image reconstruction. In addition, the data volume of the rendering parameters is much smaller than the data volume of the rendering intermediate result; therefore, even if the server of the present application also sends the rendering parameters to the terminal device, the bit rate overhead of the data stream transmitted from the server to the terminal device in the present application is also less than the bit rate overhead of the data stream transmitted in the prior art.

[0127] Exemplarily, rendering parameters may refer to all parameters that are input to the graphics rendering engine and are required for the graphics rendering engine to perform rendering processing, and may include various parameters used for rendering, the position and color vectors of all light sources, the position vectors of players or observers, the sampling method of each texture, the position coordinates of each scene object and other information, the motion trajectory of moving objects, skeletal animation parameters, etc., which are not limited in this application. Among them, the first rendering parameter and the second rendering parameter may constitute the rendering parameter (i.e., all parameters that are input to the graphics rendering engine and are required for the graphics rendering engine to perform rendering). It should be noted that the type of parameters contained in the fifth rendering parameter generated by the terminal device is the same as the type of parameters contained in the second rendering parameter generated by the central server, and the precision of the parameters contained in the fifth rendering parameter is less than or equal to the precision of the parameters contained in the second rendering parameter.

[0128] Exemplarily, the intermediate rendering result may refer to the intermediate data for generating the image / video to be displayed generated by the graphics rendering engine in the process of generating the image to be displayed (i.e., the rendered image) / video (i.e., the rendered video). Exemplarily, the intermediate rendering result may include, but is not limited to: Computer Graphics Motion Vector (CGMV), intermediate rendering image (intermediate rendering image refers to the image generated before generating the final rendered image (i.e., the rendered image mentioned above), and the computational complexity of the intermediate rendering image is lower than the computational complexity of the rendered image; wherein, the intermediate rendering image may be, for example, an intermediate rendering image without indirect lighting rendering, an intermediate rendering image without mirror reflection processing, an intermediate rendering image without highlight processing, etc.), position map, normal map, albedo map, specular intensity map, mesh ID, material ID (wherein, each material map corresponds to a Material ID), rendering ID (Render ID) (each object (or a three-dimensional object model) corresponds to a Render ID, Object ID, etc.), and the intermediate rendering result may refer to the intermediate data generated by the graphics rendering engine in the process of generating the image to be displayed (i.e., the rendered image) / video (i.e., the rendered video). ID, depth information, etc., which are not restricted in this application. Among them, the first rendering intermediate result is part of all rendering intermediate results generated in the rendering process, and the second rendering intermediate result and the third rendering intermediate result can constitute the first rendering intermediate result. It should be noted that the type of intermediate results included in the first rendering intermediate result generated by the terminal device is the same as the type of intermediate results included in the first rendering intermediate result generated by the central server, and the accuracy of the intermediate results included in the first rendering intermediate result generated by the terminal device is less than or equal to the accuracy of the intermediate results included in the first rendering intermediate result generated by the central server.

[0129] based on Figure 1 In this application scenario, this application proposes a terminal-cloud collaboration system that can be used as follows: Figure 2 shown.

[0130] Figure 2 Schematic diagram of the end-cloud collaboration system provided in an embodiment of the present application. Figure 2 The end-cloud collaborative system shown may include cloud-side devices and end-side devices.

[0131] Reference Figure 2 , for example, the cloud-side device may refer to Figure 1 The central server in the application may be a single server or a server cluster, and this application does not impose any restrictions on this.

[0132] Exemplarily, the cloud-side device may include a rendering module, an encoder, and a communication module. It should be understood that Figure 2 This is only an example of the present application. The cloud side device of the present application may include Figure 2 The present application is not limited to more or fewer modules as shown.

[0133] Exemplarily, a rendering module may be used to perform rendering processing on a three-dimensional scene based on rendering parameters to obtain a rendered image and an intermediate rendering result;

[0134] Exemplarily, the encoder may be configured to encode the rendered image based on the first rendering intermediate result, encode the second rendering intermediate result, and write the second rendering parameter into a bitstream.

[0135] Exemplarily, the communication module can be used to communicate with other electronic devices, for example, the code stream output by the encoder can be sent to the terminal side device.

[0136] Exemplarily, end-side devices include, but are not limited to: personal computers, computer workstations, smart phones, tablet computers, servers, smart cameras, smart cars or other types of cellular phones, media consumption devices, wearable devices (such as VR / AR helmets, VR glasses), set-top boxes, game consoles, etc.

[0137] Continue to refer to Figure 2 , exemplarily, the end-side device may include a communication module, a rendering module, a decoder and a display module. It should be understood that, Figure 2 This is only an example of the present application. The terminal device of the present application may include Figure 2 The present application is not limited to more or fewer modules as shown.

[0138] Exemplarily, the communication module can be used to communicate with other electronic devices, for example, to receive a code stream sent by a cloud-side device.

[0139] Exemplarily, the rendering module can be used to render the three-dimensional scene based on the rendering parameters to obtain the first rendering intermediate result or the third rendering intermediate result. It should be understood that the rendering module can also generate a rendered image, but the rendered image is not used for subsequent display module to display.

[0140] Exemplarily, the decoder may be configured to decode the code stream based on the first rendering intermediate result to obtain a reconstructed image.

[0141] Exemplarily, a display module may be used to display the reconstructed image.

[0142] It should be understood that the present application does not limit the video coding standards used by the encoder and decoder, for example, may include but is not limited to: H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), H.266 / VVC (Versatile Video Coding), AV1 (AO Media Video 1, "AO Media" refers to a video coding standard developed by the Open Media Video Alliance), etc., as well as extended standards of these video coding standards; in addition, new video coding standards and extended standards generated with the development of video coding and decoding technology may also be included.

[0143] The following is based on Figure 1 and Figure 2 , the encoding and decoding process is explained.

[0144] Figure 3 A schematic diagram of the encoding process provided in the embodiment of the present application. Figure 3 In the embodiment of the present invention, the cloud-side device is a server, the terminal-side device is a terminal device, and encoding a frame of rendered image is used as an example for description.

[0145] S301: The server renders the three-dimensional scene to obtain a rendered image.

[0146] The intermediate rendering result is generated during the above rendering process, and the intermediate rendering result includes a first intermediate rendering result.

[0147] In a possible implementation, the server may render the three-dimensional scene based on the rendering parameters, wherein the rendering parameters include first rendering parameters obtained from the terminal device.

[0148] For example, the server may obtain rendering parameters; then, the rendering parameters may be input into a graphics rendering engine, and the graphics rendering engine may render the three-dimensional scene based on the rendering parameters to obtain a rendered image. Figure 2 The rendering module in .

[0149] Exemplarily, the server may obtain a first rendering parameter from a terminal device (wherein the first rendering parameter is generated by the terminal device). In one possible manner, when the first rendering parameter generated by the terminal device is all the parameters required for the graphics rendering engine to perform rendering processing, the server may determine the first rendering parameter as the rendering parameter; that is, the rendering parameter obtained by the server may only include the first rendering parameter. In one possible manner, when the first rendering parameter generated by the terminal device is part of the parameters required for the graphics rendering engine to perform rendering processing, the server may also generate a second rendering parameter; then, the first rendering parameter and the second rendering parameter are determined as rendering parameters. That is, the rendering parameters obtained by the server may include the first rendering parameter and the second rendering parameter. The first rendering parameter and the second rendering parameter are described in detail in subsequent embodiments.

[0150] It should be noted that the server can obtain the first rendering parameter from the terminal device according to a preset period, wherein the preset period can be set as required, and this application does not limit this. It should be understood that the first rendering parameter obtained by the server from the terminal device each time can be used to render one or more frames of images.

[0151] Exemplarily, the rendering process of the graphics rendering engine can be as follows: the image engine can perform geometric transformation, projection transformation, perspective transformation and window clipping on the three-dimensional object model in the three-dimensional scene based on the input rendering parameters, and then generate a rendered image through the acquired material and light and shadow information.

[0152] In a possible implementation manner, the first rendering intermediate result may be determined according to the target operation.

[0153] Exemplarily, the graphics rendering engine in the rendering module generates a rendering intermediate result during the rendering process (the rendering process refers to the time period from the moment when the image rendering engine obtains the rendering parameters to the moment when the rendered image is obtained), and a first rendering intermediate result can be selected from the rendering intermediate results; the rendered image can be subsequently encoded based on the first rendering intermediate result.

[0154] Exemplarily, part of the rendering intermediate results can be selected from the rendering intermediate results as the first rendering intermediate result. Exemplarily, the present application may encode the rendered image based on the first rendering intermediate result by generating a virtual reference block based on the first rendering intermediate result, and then encoding the rendered image based on the virtual reference block; further, part of the rendering intermediate results that can be used to generate the virtual reference block can be selected from the rendering intermediate results as the first rendering intermediate result. For example, CGMV, a rendering intermediate image, etc. can be selected as the first rendering intermediate result, and the present application does not limit this.

[0155] S302: The server performs a target operation based on the first rendering intermediate result to generate a virtual reference block.

[0156] It can be understood that the virtual reference frame is also a virtual reference block, that is, the virtual reference frame can be regarded as a virtual reference block.

[0157] In a possible implementation manner, when a target condition is met, a target operation may be performed on a reconstructed image of the encoded image based on the first intermediate rendering result.

[0158] For example, the above target conditions may be related to the terminal device capability, target similarity, or prediction module parameters.

[0159] The target similarity is the similarity between the virtual reference block and the frame to be encoded, and the prediction module parameter is used to indicate whether to use inter-frame prediction and / or whether the virtual reference block is used in inter-frame prediction.

[0160] The terminal side device capability specifically refers to the capability of the virtual reference block supported by the terminal side device.

[0161] Exemplarily, the end-side device capabilities may include virtual reference blocks (i.e., whether the encoder of the end-side device supports the generation of virtual reference blocks), the number of virtual reference blocks supported, validity verification capability (i.e., whether validity verification is supported), non-default parameter capability (i.e., whether the use of non-default parameters to generate virtual reference blocks is supported), power consumption, latency, frame rate and other parameters.

[0162] For example, the above-mentioned goal may be satisfied by using inter-frame prediction.

[0163] For another example, the above-mentioned target may be satisfied by using a virtual reference block for inter-frame prediction.

[0164] For another example, the above-mentioned target satisfaction may be that the terminal side device capability satisfies the virtual reference frame generation requirement.

[0165] The method provided in the embodiment of the present application determines whether to execute the target operation through the target condition, thereby ensuring that the total bit rate of the end cloud is reduced while the end-side rendering screen is not affected.

[0166] In a possible implementation, the code stream includes first indication information, and when the first indication information is a first value, the code stream further includes second indication information; and / or when the second indication information is a second value, the code stream further includes third indication information.

[0167] For example, the first indication information may indicate whether the target operation is executed. The second indication information may indicate the method adopted by the target operation. The second indication information may indicate the parameters involved in the method adopted by the target operation.

[0168] For another example, the first indication information being a first value may indicate that the target step is executed, and the second indication information being a second value may indicate the method used for the target operation.

[0169] For another example, if the target operations include reprojection, validity check, weighted averaging, hole filling and intra-frame noise reduction, the first indication information may be 5 flag values, and the 5 flag values ​​are respectively used to indicate whether reprojection, validity check, weighted averaging, hole filling and intra-frame noise reduction are performed.

[0170] For another example, as shown in Table 1, in a possible implementation, the first indication information may include multiple flags, each of which may indicate whether one or more target operations are performed. For example, flag 1 is used to indicate whether reprojection is performed, flag 2 is used to indicate whether validity check is performed, flag 3 is used to indicate whether weighted averaging is performed, flag 4 is used to indicate whether hole filling is performed, and flag 5 is used to indicate whether intra-frame noise reduction is performed. When the value of flag 1 is 1, it indicates that reprojection is performed; when the value of flag 1 is 0, it indicates that reprojection is not performed.

[0171] Table 1

[0172] First instruction information Flag 1 Flag 2 Flag 3 Identification bit 4 Identification bit 5 Value 1 1 1 1 0

[0173] Exemplarily, the server performs reprojection, validity check, weighted averaging, hole filling, and intra-frame noise reduction on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block.

[0174] In a possible implementation manner, the reconstructed image may be reprojected according to the first intermediate rendering result.

[0175] Exemplarily, the pixels of the to-be-generated virtual reference block may be reprojected in the reconstructed image according to the first rendering intermediate result and according to the depth information and camera parameters in the first rendering intermediate result.

[0176] For example, the pixels of the to-be-generated virtual reference block may be reprojected in the reconstructed image according to the depth information and camera parameters in the first rendering intermediate result.

[0177] For another example, the pixels of the virtual reference block to be generated may be reprojected in the reconstructed image according to the motion vector (MV) information in the first rendering intermediate result.

[0178] In one possible implementation, the server may perform a validity check on the reprojected pixel points in the reconstructed image based on the first rendering intermediate result to obtain a validity check result, wherein the validity check result is used to indicate whether the pixel points to be detected of the virtual reference block to be generated pass the validity check, and the validity check result is used to generate the virtual reference block.

[0179] Exemplarily, based on the first rendering intermediate result, the reprojected pixel point is inversely reprojected into the virtual reference block to be generated to obtain the inversely reprojected pixel coordinates of the reprojected pixel point; based on the inversely reprojected pixel coordinates and the pixel coordinates of the pixel point to be detected, the validity check result is obtained.

[0180] For example, when the pixel coordinates of the reprojected pixel points do not exceed the resolution range of the image, the reprojected pixel points can be inversely reprojected into the virtual reference block to be generated, and the pixel coordinate error of the pixel point to be detected can be determined based on the inversely reprojected pixel coordinates and the pixel coordinates of the pixel point to be detected.

[0181] When the pixel coordinate error of the pixel to be detected is less than the error threshold, it is determined that the pixel to be detected passes the validity verification.

[0182] When the pixel coordinate error of the pixel to be detected is greater than the error threshold, it is determined that the pixel to be detected has failed the validity verification.

[0183] When the pixel coordinate error of the pixel to be detected is equal to the error threshold, it can be determined that the pixel to be detected has passed the validity verification, or it can be determined that the pixel to be detected has not passed the validity verification, which is not limited in the embodiments of the present application.

[0184] For another example, when the pixel coordinates of the reprojected pixel point exceed the resolution range of the image, it can be determined that the pixel point to be detected has not passed the validity verification.

[0185] The pixel coordinate error can be determined in any manner that can be thought of by a person skilled in the art, and the present application embodiment does not limit this. For example, the pixel coordinate error can be calculated by using Euclidean distance, Manhattan distance, variance, etc.

[0186] The error threshold can be obtained in any manner that can be thought of by a person skilled in the art, and the present application embodiment does not limit this. For example, the error threshold can be obtained from outside the codec or from the bit stream. The error threshold is also called a validity check error threshold parameter.

[0187] Also exemplarily, the above-mentioned reprojected pixel points and the above-mentioned pixel points to be detected can be projected into the world coordinate system based on the above-mentioned first rendering intermediate result to obtain the world coordinates of the above-mentioned reprojected pixel points and the world coordinates of the above-mentioned pixel points to be detected; based on the world coordinates of the above-mentioned reprojected pixel points and the world coordinates of the above-mentioned pixel points to be detected, the above-mentioned validity check result can be obtained.

[0188] For example, when the pixel coordinates of the reprojected pixel points do not exceed the resolution range of the image, the reprojected pixel points and the pixel points to be detected can be projected into the world coordinate system, and the world coordinate error of the pixel points to be detected can be determined based on the world coordinates of the reprojected pixel points and the world coordinates of the pixel points to be detected.

[0189] When the world coordinate error of the pixel to be detected is less than the error threshold, it is determined that the pixel to be detected passes the validity verification.

[0190] When the world coordinate error of the pixel to be detected is greater than the error threshold, it is determined that the pixel to be detected has failed the validity verification.

[0191] When the world coordinate error of the pixel to be detected is equal to the error threshold, it can be determined that the pixel to be detected has passed the validity verification.

[0192] For another example, when the pixel coordinates of the above-mentioned reprojected pixel point exceed the resolution range of the image, it can be determined that the pixel point to be detected has passed the validity verification, or it can be determined that the pixel point to be detected has not passed the validity verification. The embodiments of the present application are not limited to this.

[0193] The above-mentioned world coordinate error can be determined in any way that can be thought of by a person skilled in the art, and the present application embodiment does not limit this. For example, the above-mentioned world coordinate error can be calculated by using Euclidean distance, Manhattan distance, variance, etc.

[0194] Also exemplarily, the validity check result can be obtained based on the first rendering intermediate result, the target information corresponding to the reprojected image point and the target information of the pixel to be detected, wherein the target information includes at least one of an object ID, a mesh ID or a material ID.

[0195] For example, when the pixel coordinates of the reprojected pixel points do not exceed the resolution range of the image, the pixel target information of the object pointed to by the reprojected pixel points in the three-dimensional scene (such as a game) can be determined.

[0196] When the target information pointed to by the pixel to be detected is consistent with the target information pointed to by the reprojected pixel, it is determined that the pixel to be detected passes the validity verification.

[0197] If the Object IDs pointed to by the pixel to be detected and the reprojected pixel are both 357, it is determined that the pixel to be detected passes the validity verification.

[0198] When the target information pointed to by the pixel to be detected is inconsistent with the target information pointed to by the reprojected pixel, it is determined that the pixel to be detected has failed the validity verification.

[0199] If the Material ID pointed to by the pixel to be detected is 122, and the Material ID pointed to by the reprojected pixel is 155, it is determined that the pixel to be detected has failed the validity verification.

[0200] For another example, when the pixel coordinates of the reprojected pixel point exceed the resolution range of the image, it can be determined that the pixel point to be detected has not passed the validity verification.

[0201] In a possible implementation manner, one or more types of validity verifications may be selected from the above validity verifications according to the second indication information to perform validity verification on the above pixel point to be detected.

[0202] In a possible implementation manner, parameters involved in validity verification may be determined according to the third indication information.

[0203] In a possible implementation, the pixels to be detected that have not passed the validity verification may also be marked as hole pixels.

[0204] In a possible implementation, weighted averaging may be performed on the labels of the pixel points to be detected that have passed the validity verification.

[0205] In a possible implementation, the server may perform weighted averaging on the reconstructed image based on the first rendering intermediate result to obtain a weighted average result, wherein the weighted average result is used to indicate the weighted average pixel value of the virtual reference block to be generated, and the weighted average result is used to generate the virtual reference block.

[0206] Exemplarily, a weighted average is performed based on the first rendering intermediate result, the pixel values ​​of the pixels in the reconstructed image, and the image sequence number of the reconstructed image to obtain the weighted average result.

[0207] For example, in the weighted averaging process, a larger weight can be assigned to the value of the reprojected pixel point in the reconstructed image whose image sequence number is closer to the virtual reference block to be generated, and a smaller weight can be assigned to the value of the reprojected pixel point in the reconstructed image whose image sequence number is farther from the virtual reference block to be generated.

[0208] Also exemplarily, a weighted average can be performed based on the first rendering intermediate result, the pixel value of the pixel in the reconstructed image, the world coordinates of the pixel in the reconstructed image and the world coordinates of the pixel of the virtual reference block to be generated to obtain the weighted average result.

[0209] For example, in the weighted averaging process, a larger weight can be assigned to the value of the reprojected pixel point with a smaller world coordinate error with the pixel point to be weighted averaged, and a smaller weight can be assigned to the value of the reprojected pixel point with a larger world coordinate error with the pixel point to be weighted averaged.

[0210] Also exemplarily, a weighted average is performed based on the first rendering intermediate result, the pixel value of the pixel in the reconstructed image, the pixel coordinates of the pixel in the reconstructed image and the pixel coordinates of the pixel of the virtual reference block to be generated to obtain the weighted average result.

[0211] For example, the reprojected pixel points in the reconstructed image can be inversely reprojected to the virtual reference block to be generated to obtain the inversely reprojected pixel coordinates of the reprojected pixel points, and then the pixel coordinate pixel coordinate error between the pixel coordinate of the pixel point to be weighted averaged and the inversely reprojected pixel coordinates is calculated. In the weighted averaging process, a larger weight is assigned to the value of the reprojected pixel point with a smaller pixel coordinate error with the pixel point to be weighted averaged, and a smaller weight is assigned to the value of the reprojected pixel point with a larger pixel coordinate error with the pixel point to be weighted averaged.

[0212] In a possible implementation manner, one or more weighted averaging methods may be selected from the above weighted averaging methods according to the second indication information or preset parameters to perform weighted averaging on the pixels to be weighted averaged.

[0213] In a possible implementation manner, the parameters involved in the weighted average method may be determined according to the third indication information or preset parameters.

[0214] In a possible implementation manner, the pixel point to be weighted averaged may be a pixel that has passed the validity check in the virtual reference block to be generated.

[0215] In a possible implementation, the server may perform hole filling on the reconstructed image based on the first rendering intermediate result to obtain a hole filling result, wherein the hole filling result is used to indicate the pixel value of the pixel to be filled after the hole is filled, wherein the pixel to be filled is the pixel that fails the validity check in the virtual reference block to be generated, and the hole filling result is used to generate the virtual reference block.

[0216] Exemplarily, the hole filling of the reconstructed image may be performed based on the first rendering intermediate result and the reprojected pixel points of the reconstructed image to obtain the hole filling result.

[0217] For example, the pixel points to be filled can be reprojected into multiple reconstructed images to obtain multiple reprojected pixel points, and the validity of these multiple reprojected pixel points can be checked according to the pixel points to be filled. The holes of the pixel points to be filled can be filled according to the pixel values ​​of the reprojected pixel points that pass the validity check among the multiple reprojected pixel points.

[0218] For example, a weighted calculation is performed based on the pixel values ​​of the reprojected pixels that have passed the validity check to obtain the pixel values ​​of the pixel to be filled after the hole is filled.

[0219] The number of reconstructed images projected in the hole filling step may exceed the number of reconstructed images projected in the reprojection step. For example, if the number of reconstructed images projected in the reprojection step is 3 frames, the number of reconstructed images projected in the hole filling step may be greater than 3 frames.

[0220] Also exemplarily, the reconstructed image can be hole-filled based on the first rendering intermediate result and the target pixel point of the reprojected pixel point of the reconstructed image to obtain a hole-filling result, wherein the target pixel point is a pixel point in the reconstructed image whose distance from the reprojected pixel point is less than a first threshold.

[0221] For example, the pixel points to be filled can be reprojected into the reconstructed image to obtain reprojected pixel points, and then the target pixel points of the reprojected pixel points (that is, the pixel points whose distance from the above-mentioned reprojected pixel points is less than a first threshold) are checked for validity based on the pixel points to be filled. Then, the holes of the pixel points to be filled are filled based on the pixel values ​​of the pixel points among the above-mentioned target pixel points that pass the validity check.

[0222] That is, the pixel to be filled can be air-conditioned and filled by using the pixel values ​​of the pixel points around the reprojected pixel point of the pixel to be filled in the reconstructed image.

[0223] Also exemplarily, the reconstructed image may be hole-filled based on the first rendering intermediate result and non-to-be-filled pixels to obtain the hole-filling result. The non-to-be-filled pixels are pixels that have passed the validity check in the virtual reference block to be generated.

[0224] For example, non-pixels to be filled whose distances from the pixels to be filled are less than a second threshold value can be determined from the virtual reference block to be generated, and then the holes of the pixels to be filled are filled using the pixel values ​​of the non-pixels to be filled whose distances from the pixels to be filled are less than the second threshold value.

[0225] That is, the hole filling of the pixel to be filled can be performed by generating pixel values ​​of the pixel points that have passed the validity verification around the pixel point to be filled in the virtual reference block.

[0226] In a possible implementation manner, one or more methods may be selected from the above hole filling methods according to the second indication information or preset parameters to perform hole filling on the pixel points to be filled.

[0227] In a possible implementation manner, parameters involved in the hole filling method may be determined according to the third indication information or preset parameters.

[0228] S303: The server predicts the rendered image based on the virtual reference block to obtain a predicted image.

[0229] S304: The server encodes the residual image between the predicted image and the rendered image to obtain a code stream.

[0230] The method provided in the embodiment of the present application can generate a virtual reference block based on the intermediate rendering result generated in the rendering process by means of reprojection, validity check, weighted average, hole filling, intra-frame noise reduction, etc. The generated virtual reference block can provide a more accurate reference for encoding and decoding of the rendered picture, thereby improving the accuracy of inter-frame prediction, thereby improving the encoding efficiency and reducing the bit rate. In this way, it is possible to reduce the total bit rate of the end cloud while ensuring the end-side rendering of the picture.

[0231] Figure 5 A schematic diagram of the coding framework provided for an embodiment of the present application. Figure 5 The embodiment shows Figure 2 The coding frame of the encoder in . Figure 5 The video to be encoded is a rendered video and may include multiple frames of rendered images.

[0232] Reference Figure 5 Exemplarily, the encoder may include: a blocking module, an intra-frame prediction module, an inter-frame prediction module, a transform module, a quantization module, an entropy coding module, an inverse quantization module, an inverse transform module, a loop filtering module and a memory.

[0233] Exemplarily, after the rendering module generates the virtual reference block, the virtual reference block may be inserted into a reference frame list in the memory, and the length of the reference frame list may be increased by 1.

[0234] Continue to refer to Figure 5 , the encoding process of the encoder can be as follows: after the block module of the encoder obtains the video to be encoded, it can block each frame of the rendered image in the video to be encoded to obtain a block to be encoded. For each block to be encoded, the decision module ( Figure 5 (not shown) determines whether to perform inter-frame prediction or intra-frame prediction on the block to be encoded.

[0235] When it is determined to be an intra-frame prediction, the intra-frame prediction module can perform intra-frame prediction to determine the prediction block corresponding to the block to be encoded. After obtaining the prediction block, the residual block between the block to be encoded and the prediction block can be determined; then, the residual block is input into the transformation module, and the transformation module transforms the residual block to obtain the transformation result and outputs the transformation result to the quantization module. Subsequently, the quantization module can quantize the transformation result to obtain the quantization result and output the quantization result to the entropy coding module; then, the entropy coding module can entropy encode the quantization result to obtain the encoded data of the residual block, and write the encoded data of the residual block into the bitstream.

[0236] Exemplarily, the quantization module can also output the quantization result to the inverse quantization module, which performs inverse quantization to obtain the inverse quantization result and outputs the inverse quantization result to the inverse transformation module. Then, the inverse transformation module performs inverse transformation on the inverse quantization result to obtain a decoded residual block. Then, the decoded residual block can be superimposed with the prediction block output by the intra-frame prediction module to obtain a reconstructed block. Subsequently, the reconstructed block is input into the loop filtering module, which performs loop filtering on the reconstructed block to obtain a filtered reconstructed block and output the filtered reconstructed block to the memory. (Wherein, after obtaining the filtered reconstructed block, the loop filtering module can splice the filtered reconstructed block to the corresponding position of the image to be reconstructed; after obtaining a frame of reconstructed image, the reconstructed image can be output to the memory for storage).

[0237] When it is determined that the prediction is inter-frame prediction, the inter-frame prediction module may perform inter-frame prediction to determine the prediction block corresponding to the block to be encoded.

[0238] In one possible approach, the inter-frame prediction module may select a virtual reference block from a reference frame sequence in a memory as a reference frame of the rendered image. For a block to be encoded in the rendered image, a matching prediction block (that is, the above-mentioned prediction image) may be searched from the virtual reference block.

[0239] In one possible manner, the inter-frame prediction module can use both the virtual reference block and the original reference frame from the reference frame sequence in the memory as candidate reference frames for the rendered image; wherein the original reference frame refers to the reconstructed image. For the block to be encoded in the rendered image, inter-frame prediction can be performed based on multiple candidate reference frames to determine multiple prediction blocks; wherein one candidate reference frame corresponds to one prediction block; and then an optimal prediction block can be selected from the multiple prediction blocks (for example, the rate-distortion cost can be used as an evaluation criterion, and the prediction block with the smallest rate-distortion cost can be determined as the optimal prediction block (that is, the above-mentioned prediction image)).

[0240] After obtaining the prediction block, the residual block between the block to be encoded and the prediction block (that is, the residual image mentioned above) can be determined; then, the residual block is input into the transformation module, which transforms the residual block to obtain the transformation result and outputs the transformation result to the quantization module. Subsequently, the quantization module can quantize the transformation result to obtain the quantization result and output the quantization result to the entropy coding module; then, the entropy coding module can entropy encode the quantization result to obtain the coded data of the residual block, and write the coded data of the residual block into the bitstream (that is, the bitstream obtained in S305 above).

[0241] Exemplarily, the quantization module can also output the quantization result to the inverse quantization module, which performs inverse quantization to obtain the inverse quantization result and outputs the inverse quantization result to the inverse transformation module. Then, the inverse transformation module performs inverse transformation on the inverse quantization result to obtain a decoded residual block. Then, the decoded residual block can be superimposed with the prediction block output by the intra-frame prediction module to obtain a reconstructed block. Subsequently, the reconstructed block is input into the loop filtering module, which performs loop filtering on the reconstructed block to obtain a filtered reconstructed block and output the filtered reconstructed block to the memory. (Wherein, after obtaining the filtered reconstructed block, the loop filtering module can splice the filtered reconstructed block to the corresponding position of the image to be reconstructed; after obtaining a frame of reconstructed image, the reconstructed image can be output to the memory for storage).

[0242] Exemplarily, after the rendered image is encoded, the virtual reference block in the reference frame list in the memory may be deleted, and the length of the reference frame list may be reduced by 1, so as to avoid the reference frame list being too long and reducing the encoding efficiency.

[0243] Reference Figure 6 , exemplarily, the above encoder may further include: a virtual reference block generation module.

[0244] The virtual reference block generation module is used to generate a virtual reference block and send the generated virtual reference block to the inter-frame prediction module (the motion estimation module and the motion compensation module in the inter-frame prediction module) for subsequent video encoding.

[0245] In a possible implementation, the virtual reference block generation module may also be integrated with other modules in the encoder.

[0246] For example, the virtual reference block generation module can be integrated into the inter-frame prediction module.

[0247] The specific method in which the virtual reference block generation module generates the virtual reference block can refer to the above step S303, which will not be repeated here.

[0248] It can be understood that by placing the virtual reference block generation module that generates the virtual reference block into the codec, on some devices that harden the codec, the speed of generating the virtual reference block is improved while ensuring the reduction of the total bit rate of the end cloud.

[0249] Figure 7 A schematic diagram of the decoding process provided in an embodiment of the present application. Figure 7 Shown is Figure 3 The corresponding decoding process. Figure 7 In an embodiment of the present invention, the terminal device performs rendering processing based on locally generated rendering parameters and generates a first rendering intermediate result. Figure 7 In the embodiment, decoding of one frame of image is taken as an example for description.

[0250] S701, the terminal device receives a code stream.

[0251] Exemplarily, after the server sends the code stream, the terminal device can receive the code stream.

[0252] In a possible implementation, the terminal-side device information may be sent to the cloud-side device.

[0253] In one possible implementation, indication information is received from a cloud-side device, where the indication information includes first indication information, and when the first indication information is a first value, the indication information also includes second indication information; and / or when the second indication information is a second value, the indication information also includes third indication information.

[0254] S702: The terminal device decodes the code stream to obtain a residual image of the current image.

[0255] In a possible implementation, the above-mentioned code stream can be decoded to obtain a residual image of the current image; the above-mentioned code stream is decoded to obtain indication information, the above-mentioned indication information includes first indication information, and when the first indication information is a first value, the above-mentioned indication information also includes second indication information; and / or, when the second indication information is a second value, the above-mentioned indication information also includes third indication information.

[0256] Figure 8 A schematic diagram of a decoding framework provided in an embodiment of the present application. Figure 8 The embodiment shows Figure 2 The decoding framework of the decoder.

[0257] Reference Figure 8 Exemplarily, the decoder may include: an entropy decoding module, an inverse quantization module, an inverse transformation module, an intra-frame prediction module, an inter-frame prediction module, a loop filtering module and a memory.

[0258] Exemplarily, after receiving the code stream, the terminal device can parse the code stream to obtain the parsing result. The parsing process can be as follows: extract the encoded data of the residual block from the code stream, and then input the encoded data of the residual block into the entropy decoding module, and the entropy decoding module performs entropy decoding on the encoded data of the residual block to obtain entropy decoded data. Then, the entropy decoded data can be input into the inverse quantization module, and the inverse quantization module performs inverse quantization to obtain inverse quantized data and input the inverse quantized data into the inverse transformation module. Subsequently, the inverse transformation module performs inverse transformation on the inverse quantized data to obtain the decoded residual block (that is, the residual image corresponding to the current frame in S702).

[0259] It should be understood that when the encoder performs lossy encoding on the residual block, the residual block obtained by the decoder is different from the residual block encoded by the encoder. When the encoder performs lossless encoding on the residual block, the residual block obtained by the decoder is the same as the residual block encoded by the encoder.

[0260] It should be noted that the code stream description information for describing the code stream can also be extracted from the code stream, in which case there is no need to perform entropy decoding, inverse quantization and inverse transformation on the code stream description information. The code stream description information can be used in the subsequent decoding process.

[0261] S703: The terminal device renders the three-dimensional scene to obtain a rendered image.

[0262] The intermediate rendering result is generated during the above rendering process, and the intermediate rendering result includes a first intermediate rendering result.

[0263] In a possible implementation manner, the first rendering intermediate result is determined according to the target operation.

[0264] In a possible implementation, the terminal device may perform rendering processing on the three-dimensional scene based on rendering parameters corresponding to the current frame, and generate a first rendering intermediate result during the rendering process; wherein the rendering parameters include first rendering parameters generated by the terminal device.

[0265] Exemplarily, after parsing and obtaining the residual image corresponding to the current frame, the rendering module (which may be a graphics rendering engine included in the rendering module) may render the three-dimensional scene based on the rendering parameters corresponding to the current frame.

[0266] Exemplarily, when the first rendering parameters uploaded by the terminal device to the server are all the parameters required by the graphics rendering engine for rendering processing, the rendering module can render the three-dimensional scene based on the first rendering parameters corresponding to the current frame. When the first rendering parameters uploaded by the terminal device are part of the parameters required by the graphics rendering engine for rendering processing, the terminal device can also generate a fifth rendering parameter, and then the rendering module can render the three-dimensional scene based on the first rendering parameters and the fifth rendering parameters corresponding to the current frame.

[0267] It should be noted that the type of parameters included in the fifth rendering parameter generated by the terminal device is the same as the type of parameters included in the second rendering parameter generated by the central server; the precision of the parameters included in the fifth rendering parameter is less than or equal to the precision of the parameters included in the second rendering parameter.

[0268] It should be noted that the type of intermediate results contained in the first rendering intermediate result generated by the terminal device is the same as the type of intermediate results contained in the first rendering intermediate result generated by the server; the precision of the intermediate results contained in the first rendering intermediate result generated by the terminal device is less than or equal to the precision of the intermediate results contained in the first rendering intermediate result generated by the server.

[0269] It should be noted that the graphics rendering engine of the terminal device can also obtain a rendered image by rendering processing; since the computing power of the terminal device is lower than that of the server, the image quality of the rendered image obtained by the graphics rendering engine of the terminal device through rendering processing (wherein the image quality may include subjective quality and objective quality (objective quality such as rate-distortion performance, PSNR (Peak Signal to Noise Ratio)) is lower than the image quality of the rendered image obtained by the graphics rendering engine of the server through rendering processing; therefore, the terminal device may not display the rendered image obtained by the graphics rendering engine of the terminal device through rendering processing, but may display the reconstructed image obtained by decoding the code stream.

[0270] S704: Perform a target operation on the reconstructed image based on the first intermediate rendering result to generate a virtual reference block.

[0271] The target operation includes at least one of reprojection, validity check, weighted average, hole filling or intra-frame noise reduction.

[0272] In a possible implementation, the reprojected pixel points in the reconstructed image can be validated based on the first rendering intermediate result to obtain a validity test result. The validity test result is used to indicate whether the pixel points to be detected of the virtual reference block to be generated have passed the validity test. The validity test result is used to generate the virtual reference block.

[0273] In a possible implementation, the reprojected pixel point can be inversely reprojected into the virtual reference block to be generated based on the first rendering intermediate result to obtain the inversely reprojected pixel coordinates of the reprojected pixel point; and the validity check result can be obtained based on the inversely reprojected pixel coordinates and the pixel coordinates of the pixel point to be detected.

[0274] In a possible implementation method, the above-mentioned reprojected pixel points and the above-mentioned pixel points to be detected can be projected into the world coordinate system based on the above-mentioned first rendering intermediate result to obtain the world coordinates of the above-mentioned reprojected pixel points and the world coordinates of the above-mentioned pixel points to be detected; based on the world coordinates of the above-mentioned reprojected pixel points and the world coordinates of the above-mentioned pixel points to be detected, the above-mentioned validity check result can be obtained.

[0275] In a possible implementation, the validity check result can be obtained based on the first rendering intermediate result, the target information corresponding to the reprojected image point and the target information of the pixel to be detected, wherein the target information includes at least one of an object ID, a mesh ID or a material ID.

[0276] In a possible implementation, the reconstructed image may be weighted averaged based on the first rendering intermediate result to obtain a weighted average result, wherein the weighted average result is used to indicate the weighted average pixel value of the virtual reference block to be generated, and the weighted average result is used to generate the virtual reference block.

[0277] In a possible implementation, a weighted average may be performed based on the first rendering intermediate result, the pixel values ​​of the pixels in the reconstructed image, and the image sequence number of the reconstructed image to obtain the weighted average result.

[0278] In a possible implementation, a weighted average may be performed based on the first rendering intermediate result, the pixel value of the pixel in the reconstructed image, the world coordinates of the pixel in the reconstructed image, and the world coordinates of the pixel of the virtual reference block to be generated to obtain the weighted average result.

[0279] In a possible implementation, a weighted average may be performed based on the first rendering intermediate result, the pixel value of the pixel in the reconstructed image, the pixel coordinates of the pixel in the reconstructed image, and the pixel coordinates of the pixel of the virtual reference block to be generated to obtain the weighted average result.

[0280] In a possible implementation, holes in the reconstructed image can be filled based on the first rendering intermediate result to obtain a hole filling result, wherein the hole filling result is used to indicate the pixel value of the pixel to be filled after the hole is filled, and the pixel to be filled is the pixel that fails the validity check in the virtual reference block to be generated, and the hole filling result is used to generate the virtual reference block.

[0281] In a possible implementation, holes in the reconstructed image may be filled based on the first rendering intermediate result and the reprojected pixel points of the reconstructed image to obtain the hole filling result.

[0282] In a possible implementation, the reconstructed image can be hole-filled based on the first rendering intermediate result and the target pixel point of the reprojected pixel point of the reconstructed image to obtain a hole-filling result, wherein the target pixel point is a pixel point in the reconstructed image whose distance from the reprojected pixel point is less than a first threshold.

[0283] In a possible implementation, holes in the reconstructed image may be filled based on the first rendering intermediate result and non-to-be-filled pixels to obtain the hole filling result. The non-to-be-filled pixels are pixels that have passed the validity check in the virtual reference block to be generated.

[0284] Exemplarily, after generating the virtual reference block, the terminal device may reconstruct the residual image corresponding to the current frame based on the virtual reference block to obtain a reconstructed image. The image quality of the reconstructed image is higher than the rendered image obtained by the graphics rendering engine of the terminal device. Subsequently, the terminal device may display the reconstructed image.

[0285] It should be noted that S701 to S703 may be executed by a rendering module in the terminal device, S704 may be executed by a rendering module in the terminal device, or may be executed by a rendering module in the terminal device. Figure 2 The following S705 to S706 can be executed by other modules other than the modules shown in the figure (the present application takes the rendering module executing S704 as an example for explanation). Figure 2 The decoder in is executed.

[0286] S705: The terminal device predicts the current frame based on the virtual reference block to obtain a predicted image.

[0287] S706: The terminal device reconstructs based on the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0288] Refer again Figure 8 , exemplarily, after generating the virtual reference block, the rendering module may insert the virtual reference block into a reference frame list in the memory and increase the length of the reference frame list by 1.

[0289] Refer again Figure 8 Exemplarily, when it is determined that it is an intra-frame prediction based on the code stream description information parsed from the code stream, the intra-frame prediction module can perform intra-frame prediction to obtain a prediction block. Afterwards, the decoded residual block can be superimposed with the prediction block to obtain a reconstructed block. Then, the reconstructed block can be loop filtered to obtain a filtered reconstructed block and the filtered reconstructed block can be output to a memory. (After obtaining the filtered reconstructed block, the loop filtering module can splice the filtered reconstructed block to the corresponding position of the image to be reconstructed; after obtaining a frame of reconstructed image, the reconstructed image can be output to the memory for storage).

[0290] Refer again Figure 8 For example, when it is determined that inter-frame prediction is performed according to the code stream description information parsed from the code stream, the inter-frame prediction module can perform inter-frame prediction to obtain a prediction block (that is, the prediction image in the above S705).

[0291] In one possible approach, when the inter-frame prediction module determines that the reference frame is a virtual reference block based on the code stream description information parsed from the code stream, the inter-frame prediction module can select the corresponding virtual reference block from the reference frame sequence in the memory as the reference frame; then, the prediction block can be determined based on the virtual reference block.

[0292] In one possible approach, when the inter-frame prediction module determines that the reference frame is the original reference frame from the code stream description information parsed from the code stream, the inter-frame prediction module can select the corresponding original reference frame from the reference frame sequence in the memory as the reference frame; then, the prediction block can be determined based on the original reference frame.

[0293] After obtaining the prediction block, the decoded residual block can be superimposed with the prediction block to obtain a reconstructed block. Then, the reconstructed block can be loop filtered to obtain a filtered reconstructed block and the filtered reconstructed block can be output to the memory. (After obtaining the filtered reconstructed block, the loop filtering module can splice the filtered reconstructed block to the corresponding position of the image to be reconstructed; after obtaining a frame of reconstructed image, the reconstructed image can be output to the memory).

[0294] It should be understood that the terminal device can also generate a code stream based on the first rendering parameter, and then send it to the server according to a preset period. In this way, the server can obtain the first rendering parameter. In addition, since the data volume of the first rendering parameter is small, the terminal device can encode the first rendering parameter and write the encoded data of the first rendering parameter into the code stream before sending it to the server; the terminal device may not encode the first rendering parameter, but directly write the first rendering parameter into the code stream and send it to the server; this application does not limit this.

[0295] In this way, the present application performs all rendering on the terminal device, so that the server does not need to send the intermediate rendering results to the terminal device; therefore, the present application can reduce the interaction delay while ensuring that the bit rate overhead of the data stream transmitted from the server to the terminal device is effectively reduced. In addition, since the intermediate rendering results are highly correlated with the rendered image, the present application encodes the rendered image based on the intermediate rendering results to ensure the quality of image reconstruction.

[0296] Reference Fig. 9 , exemplarily, the above decoder may further include: a virtual reference block generation module.

[0297] The virtual reference block generation module is used to generate a virtual reference block and send the generated virtual reference block to the inter-frame prediction module (the motion estimation module and the motion compensation module in the inter-frame prediction module) for subsequent video encoding.

[0298] In a possible implementation, the virtual reference block generation module may also be integrated with other modules in the decoder.

[0299] For example, the virtual reference block generation module can be integrated into the inter-frame prediction module.

[0300] The specific method in which the virtual reference block generation module generates the virtual reference block can refer to the above step S704, which will not be repeated here.

[0301] It can be understood that by placing the virtual reference block generation module that generates the virtual reference block into the codec, on some devices that harden the codec, the speed of generating the virtual reference block is improved while ensuring the reduction of the total bit rate of the end cloud.

[0302] An embodiment of the present application also provides a chip, which may be the chip of the cloud-side device mentioned above or the chip of the terminal-side device mentioned above. Fig.10 1 shows a schematic diagram of the structure of a chip 1000. The chip 1000 includes one or more processors 1001 and an interface circuit 1002. Optionally, the chip 1000 may also include a bus 1003.

[0303] The processor 1001 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above encoding and decoding method may be completed by an integrated logic circuit of hardware in the processor 1001 or by instructions in the form of software.

[0304] Optionally, the processor 1001 may be a general purpose processor, a digital signal processing (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods and steps disclosed in the embodiments of the present application may be implemented or executed. The general purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0305] The interface circuit 1002 can be used to send or receive data, instructions or information. The processor 1001 can use the data, instructions or other information received by the interface circuit 1002 to process, and can send the processing completion information through the interface circuit 1002.

[0306] Optionally, the chip also includes a memory, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).

[0307] Optionally, the memory stores executable software modules or data structures, and the processor can perform corresponding operations by calling operation instructions stored in the memory (the operation instructions can be stored in the operating system).

[0308] Optionally, the chip can be used in the electronic device involved in the embodiment of the present application. Optionally, the interface circuit 1002 can be used to output the execution result of the processor 1001. The encoding and decoding method provided in one or more embodiments of the embodiment of the present application can refer to the aforementioned embodiments, which will not be repeated here.

[0309] It should be noted that the corresponding functions of the processor 1001 and the interface circuit 1002 can be implemented through hardware design, software design, or a combination of hardware and software, and there is no limitation here.

[0310] Fig.11This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may be the cloud-side device, the processor of the cloud-side device, or a chip in the processor of the cloud-side device, or a functional module of the cloud-side device. The electronic device may also be the terminal-side device, the processor of the terminal-side device, or a chip in the processor of the terminal-side device, or a functional module of the terminal-side device. Fig.11 As shown, the electronic device 1100 includes a processor 1101 , a transceiver 1102 and a communication line 1103 .

[0311] Among them, the processor 1101 is used to execute any step in the encoding and decoding method provided in the embodiment of the present application, and in the process of executing any step in the encoding and decoding method provided in the embodiment of the present application, the transceiver 1102 and the communication line 1103 can be selectively called to complete the corresponding operation.

[0312] Furthermore, the electronic device 1100 may also include a memory 1104. The processor 1101, the memory 1104 and the transceiver 1102 may be connected via a communication line 1103.

[0313] The processor 1101 is a processor, a general-purpose processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 1101 may also be other devices with processing functions, such as circuits, devices, or software modules, without limitation.

[0314] The transceiver 1102 is used to communicate with other devices or other communication networks, and other communication networks may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The transceiver 1102 may be a module, a circuit, a transceiver, or any device capable of achieving communication.

[0315] The transceiver 1102 is mainly used for sending and receiving commands and information, etc., and may include a transmitter and a receiver, which respectively send and receive commands and information, etc.; operations other than sending and receiving commands and information, etc. are implemented by the processor.

[0316] The communication line 1103 is used to transmit information between the components included in the electronic device 1100.

[0317] In one design, the processor may be considered as the logic circuit and the transceiver may be considered as the interface circuit.

[0318] The memory 1104 is used to store instructions, where the instructions may be computer programs.

[0319] The memory 1104 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. The nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DRRAM). The memory 1104 may also be a compact disc read-only memory (CD-ROM) or other optical disk storage, optical disk storage (including compact disc, laser disc, optical disk, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, etc. It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0320] It should be noted that the memory 1104 can exist independently of the processor 1101, or can be integrated with the processor 1101. The memory 1104 can be used to store instructions or program codes or some data, etc. The memory 1104 can be located inside the electronic device 1100, or outside the electronic device 1100, without limitation. The processor 1101 is used to execute the instructions stored in the memory 1104 to implement the method provided in the above embodiment of the present application.

[0321] In one example, the processor 1101 may include one or more processors, such as Fig.11Processor 0 and processor 1 in.

[0322] As an optional implementation, the electronic device 1100 includes multiple processors, for example, Fig.11 In addition to the processor 1101, a processor 1107 may also be included.

[0323] As an optional implementation, the electronic device 1100 further includes an output device 1105 and an input device 1106. Exemplarily, the input device 1106 is a keyboard, a mouse, a microphone, a joystick, and the like, and the output device 1105 is a display screen, a speaker, and the like.

[0324] It should be noted that the electronic device 1100 may be a chip system or a Fig.11 Devices with similar structures in the chip system. Among them, the chip system can be composed of chips, or it can include chips and other discrete devices. The actions, terms, etc. involved in the various embodiments of this application can refer to each other without limitation. The message name or parameter name in the message exchanged between the various devices in the embodiment of this application is only an example, and other names can also be used in the specific implementation without limitation. In addition, Fig.11 The components shown in the figure do not constitute a limitation on the electronic device 1100. Fig.11 In addition to the components shown, the electronic device 1100 may include Fig.11 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0325] The processor and transceiver described in the present application can be implemented in an integrated circuit (IC), an analog IC, a radio frequency integrated circuit, a mixed signal IC, an application specific integrated circuit (ASIC), a printed circuit board (PCB), an electronic device, etc. The processor and transceiver can also be manufactured using various IC process technologies, such as complementary metal oxide semiconductor (CMOS), N-type metal oxide semiconductor (NMOS), P-type metal oxide semiconductor (positive channel metal oxide semiconductor, PMOS), bipolar junction transistor (BJT), bipolar CMOS (BiCMOS), silicon germanium (SiGe), gallium arsenide (GaAs), etc.

[0326] Fig.12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may be the cloud-side device, the processor of the cloud-side device, or a chip in the processor of the cloud-side device, or a functional module of the cloud-side device. The electronic device may also be the terminal-side device, the processor of the terminal-side device, or a chip in the processor of the terminal-side device, or a functional module of the terminal-side device. The electronic device may be applicable to the scenario shown in the above method embodiment. For ease of explanation, Fig.12 Only the main components of the electronic device are shown, including a processor 1201, a memory 1202, a control circuit 1203, and an input-output device 1204. The processor 1201 is mainly used to process the communication protocol and communication data, execute the software program, and process the data of the software program. The memory 1202 is mainly used to store the software program and data. The control circuit 1203 is mainly used for power supply and transmission of various electrical signals. The input-output device 1204 is mainly used to receive data input by the user and output data to the user.

[0327] When the electronic device is a processor 1201, the control circuit 1203 can be a mainboard, the memory 1202 includes a hard disk, RAM, ROM and other media with storage functions, the processor 1201 can include a baseband processor 1201 and a central processing unit, the baseband processor is mainly used to process the communication protocol and communication data, the central processing unit is mainly used to control the entire electronic device, execute software programs, and process the data of the software programs, and the input and output device 1204 includes a display screen, a keyboard and a mouse, etc.; the control circuit 1203 can further include or connect a transceiver circuit or a transceiver, such as: a network cable interface, etc., for sending or receiving data or signals, such as data transmission and communication with other devices. Further, it can also include an antenna for sending and receiving wireless signals, and for data / signal transmission with other devices.

[0328] The embodiment of the present application also provides a coding device, including a rendering module, an encoder and a communication module. The rendering module is used to render a three-dimensional scene to obtain a rendered image, and the rendering intermediate result is generated during the rendering process, and the rendering intermediate result includes a first rendering intermediate result; the rendering module is also used to perform a target operation on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block, and the target operation includes at least one of reprojection, validity check, weighted average, hole filling or intra-frame noise reduction; the encoder is used to predict the rendered image based on the virtual reference block to obtain a predicted image; the encoder is also used to encode the residual image between the predicted image and the rendered image to obtain a code stream; the communication module is used to send the code stream.

[0329] The embodiment of the present application also provides a decoding device, including a rendering module, a decoder and a communication module. The communication module is used to receive a code stream. The decoder is used to decode the code stream to obtain a residual image of the current image. The rendering module is used to render a three-dimensional scene to obtain a rendered image, wherein the rendering intermediate result is generated during the rendering process, and the rendering intermediate result includes a first rendering intermediate result. The rendering module is also used to perform a target operation on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block, and the target operation includes at least one of reprojection, validity check, weighted average, hole filling or intra-frame noise reduction. The decoder is also used to predict the rendered image based on the virtual reference block to obtain a predicted image. The decoder is also used to reconstruct based on the predicted image and the residual image to obtain a reconstructed image of the current frame.

[0330] An embodiment of the present application also provides a terminal-cloud collaborative system, including any encoding device provided in the embodiment of the present application and any decoding device provided in the embodiment of the present application.

[0331] An embodiment of the present application provides a cloud-side device, including: a memory and a processor, the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the server executes any one of the encoding methods provided in the embodiment of the present application.

[0332] An embodiment of the present application provides a terminal device, including: a memory and a processor, the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the terminal device executes any one of the decoding methods provided in the embodiments of the present application.

[0333] The present application embodiment provides an encoder, including a processing circuit, for executing any encoding method provided in the present application embodiment

[0334] The present application embodiment provides a decoder, including a processing circuit, for executing any one of the decoding methods provided in the present application embodiment.

[0335] The present application provides a code stream generation method, which generates a code stream based on a virtual reference block provided in an embodiment of the present application.

[0336] An embodiment of the present application provides a device for storing a code stream, the device comprising: a receiver and at least one storage medium, the receiver is used to receive the code stream; the at least one storage medium is used to store the code stream; the code stream is generated according to any one of the methods provided in the embodiments of the present application.

[0337] An embodiment of the present application provides a device for transmitting a code stream, the device comprising: a transmitter and at least one storage medium, the at least one storage medium is used to store the code stream, and the code stream is generated according to any one of the methods provided in the embodiments of the present application; the transmitter is used to obtain the code stream from the storage medium and send the code stream to the terminal side device through the transmission medium.

[0338] An embodiment of the present application provides a system for distributing code streams, the system comprising: at least one storage medium for storing at least one code stream, wherein the at least one code stream is generated according to any one of the methods provided in the embodiments of the present application, and a streaming media device for obtaining a target code stream from the at least one storage medium and sending the target code stream to a terminal device, wherein the streaming media device comprises a content server or a content distribution server.

[0339] An embodiment of the present application also provides an electronic device, which includes: at least one processor, when the at least one processor executes program code or instructions, the above-mentioned related method steps are implemented to implement the encoding and decoding method in the above-mentioned embodiment.

[0340] Optionally, the device may further include at least one memory, and the at least one memory is used to store the program code or instruction.

[0341] An embodiment of the present application also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the encoding and decoding method in the above-mentioned embodiment.

[0342] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the encoding and decoding method in the above-mentioned embodiment.

[0343] The embodiment of the present application also provides an electronic device, which may be a chip, an integrated circuit, a component or a module. Specifically, the device may include a connected processor and a memory for storing instructions, or the device includes at least one processor for obtaining instructions from an external memory. When the device is running, the processor may execute instructions so that the chip executes the encoding and decoding methods in the above-mentioned method embodiments.

[0344] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0345] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0346] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0347] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0348] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0349] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0350] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the above methods in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0351] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A coding method, characterized in that: Applied to a cloud-side device, the method includes: Rendering the three-dimensional scene to obtain a rendered image, wherein an intermediate rendering result is generated during the rendering process, and the intermediate rendering result includes a first intermediate rendering result; Performing a target operation on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block, wherein the target operation includes at least one of reprojection, validity check, weighted average, hole filling, or intra-frame noise reduction; Predicting the rendered image based on the virtual reference block to obtain a predicted image; A residual image between the predicted image and the rendered image is encoded to obtain a code stream.

2. The method according to claim 1, characterized in that The method further comprises: When the target condition is met, a target operation is performed on the reconstructed image of the encoded image based on the first rendering intermediate result.

3. The method according to claim 1 or 2, characterized in that: The code stream includes first indication information, and when the first indication information is a first value, the code stream further includes second indication information; And / or, when the second indication information is a second value, the code stream further includes third indication information.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: The first rendering intermediate result is determined according to the target operation.

5. The method according to any one of claims 1 to 4, characterized in that: The target operation includes a validity check, and the target operation is performed on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block, including: Based on the first rendering intermediate result, the reprojected pixel points in the reconstructed image are subjected to validity check to obtain a validity check result, wherein the validity check result is used to indicate whether the pixel points to be detected of the virtual reference block to be generated pass the validity check, and the validity check result is used to generate the virtual reference block.

6. The method according to claim 5, characterized in that The performing validity check on the reprojected pixel points in the reconstructed image based on the first rendering intermediate result to obtain a validity check result includes: Based on the first rendering intermediate result, inversely reprojecting the reprojected pixel point into the to-be-generated virtual reference block to obtain inversely reprojected pixel coordinates of the reprojected pixel point; The validity check result is obtained based on the inverse reprojection pixel coordinates and the pixel coordinates of the pixel point to be detected.

7. The method according to claim 5 or 6, characterized in that: The performing validity check on the reprojected pixel points in the reconstructed image based on the first rendering intermediate result to obtain a validity check result includes: Projecting the reprojected pixel point and the pixel point to be detected into a world coordinate system based on the first rendering intermediate result to obtain the world coordinates of the reprojected pixel point and the world coordinates of the pixel point to be detected; The validity check result is obtained based on the world coordinates of the reprojected pixel point and the world coordinates of the pixel point to be detected.

8. The method according to any one of claims 5 to 7, characterized in that The performing validity check on the reprojected pixel points in the reconstructed image based on the first rendering intermediate result to obtain a validity check result includes: The validity check result is obtained based on the first rendering intermediate result, the target information corresponding to the reprojected image point and the target information of the pixel point to be detected, wherein the target information includes at least one of an object identifier Object ID, a mesh identifier Mesh ID or a material identifier Material ID.

9. The method according to any one of claims 1 to 8, characterized in that The target operation includes weighted averaging, and performing the target operation on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block includes: The reconstructed image is weighted averaged based on the first rendering intermediate result to obtain a weighted average result, wherein the weighted average result is used to indicate a weighted average pixel value of a virtual reference block to be generated, and the weighted average result is used to generate the virtual reference block.

10. The method according to claim 9, characterized in that The performing weighted averaging on the reconstructed image based on the first rendering intermediate result to obtain a weighted average result includes: A weighted average is performed based on the first rendering intermediate result, the pixel values ​​of the pixels in the reconstructed image, and the image sequence number of the reconstructed image to obtain the weighted average result.

11. The method according to claim 9 or 10, characterized in that: The performing weighted averaging on the reconstructed image based on the first rendering intermediate result to obtain a weighted average result includes: A weighted average is performed based on the first rendering intermediate result, the pixel value of the pixel in the reconstructed image, the world coordinates of the pixel in the reconstructed image and the world coordinates of the pixel of the virtual reference block to be generated to obtain the weighted average result.

12. The method according to any one of claims 9 to 11, characterized in that The performing weighted averaging on the reconstructed image based on the first rendering intermediate result to obtain a weighted average result includes: A weighted average is performed based on the first rendering intermediate result, the pixel value of the pixel point in the reconstructed image, the pixel coordinates of the pixel point in the reconstructed image and the pixel coordinates of the pixel point of the virtual reference block to be generated to obtain the weighted average result.

13. The method according to any one of claims 1 to 12, characterized in that The target operation includes hole filling, and the target operation is performed on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block, including: Based on the first rendering intermediate result, holes are filled in the reconstructed image to obtain a hole filling result, wherein the hole filling result is used to indicate the pixel value of the pixel to be filled after the hole is filled, and the pixel to be filled is the pixel that fails the validity check in the virtual reference block to be generated, and the hole filling result is used to generate the virtual reference block.

14. The method according to claim 13, characterized in that The performing hole filling on the reconstructed image based on the first rendering intermediate result to obtain a hole filling result includes: Holes of the reconstructed image are filled based on the first rendering intermediate result and the reprojected pixels of the reconstructed image to obtain the hole filling result.

15. The method according to claim 13 or 14, characterized in that The performing hole filling on the reconstructed image based on the first rendering intermediate result to obtain a hole filling result includes: The reconstructed image is hole-filled based on the first rendering intermediate result and the target pixel point of the reprojected pixel point of the reconstructed image to obtain a hole-filling result, wherein the target pixel point is a pixel point in the reconstructed image whose distance from the reprojected pixel point is less than a first threshold.

16. The method according to any one of claims 13 to 15, characterized in that The performing hole filling on the reconstructed image based on the first rendering intermediate result to obtain a hole filling result includes: Holes are filled in the reconstructed image based on the first rendering intermediate result and non-to-be-filled pixels to obtain the hole filling result, where the non-to-be-filled pixels are pixels that pass the validity check in the virtual reference block to be generated.

17. A decoding method, characterized in that: Applied to a terminal device, the method includes: Receive code stream; Decoding the code stream to obtain a residual image of the current image; Rendering the three-dimensional scene to obtain a rendered image, wherein an intermediate rendering result is generated during the rendering process, and the intermediate rendering result includes a first intermediate rendering result; Performing a target operation on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block, wherein the target operation includes at least one of reprojection, validity check, weighted average, hole filling, or intra-frame noise reduction; Predicting the rendered image based on the virtual reference block to obtain a predicted image; Reconstruction is performed based on the predicted image and the residual image to obtain a reconstructed image of the current frame.

18. The method according to claim 17, characterized in that The method further comprises: Send the end-side device information to the cloud-side device.

19. The method according to claim 17 or 18, characterized in that The decoding process of the code stream comprises: Decoding the code stream to obtain a residual image of the current image; Decoding the code stream to obtain indication information, where the indication information includes first indication information, and when the first indication information is a first value, the indication information further includes second indication information; And / or, when the second indication information is the second value, the indication information further includes third indication information.

20. The method according to any one of claims 17 to 19, characterized in that The method further comprises: Receiving indication information from a cloud-side device, the indication information including first indication information, and when the first indication information is a first value, the indication information further includes second indication information; And / or, when the second indication information is the second value, the indication information further includes third indication information.

21. The method according to any one of claims 17 to 20, characterized in that The method further comprises: The first rendering intermediate result is determined according to the target operation.

22. The method according to any one of claims 17 to 21, characterized in that The target operation includes a validity check, and the target operation is performed on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block, including: Based on the first rendering intermediate result, the reprojected pixel points in the reconstructed image are subjected to validity check to obtain a validity check result, wherein the validity check result is used to indicate whether the pixel points to be detected of the virtual reference block to be generated pass the validity check, and the validity check result is used to generate the virtual reference block.

23. The method according to claim 22, characterized in that The performing validity check on the reprojected pixel points in the reconstructed image based on the first rendering intermediate result to obtain a validity check result includes: Based on the first rendering intermediate result, inversely reprojecting the reprojected pixel point into the to-be-generated virtual reference block to obtain inversely reprojected pixel coordinates of the reprojected pixel point; The validity check result is obtained based on the inverse reprojection pixel coordinates and the pixel coordinates of the pixel point to be detected.

24. The method according to claim 22 or 23, characterized in that The performing validity check on the reprojected pixel points in the reconstructed image based on the first rendering intermediate result to obtain a validity check result includes: Projecting the reprojected pixel point and the pixel point to be detected into a world coordinate system based on the first rendering intermediate result to obtain the world coordinates of the reprojected pixel point and the world coordinates of the pixel point to be detected; The validity check result is obtained based on the world coordinates of the reprojected pixel point and the world coordinates of the pixel point to be detected.

25. The method according to any one of claims 22 to 24, characterized in that The performing validity check on the reprojected pixel points in the reconstructed image based on the first rendering intermediate result to obtain a validity check result includes: The validity check result is obtained based on the first rendering intermediate result, the target information corresponding to the reprojected image point and the target information of the pixel point to be detected, wherein the target information includes at least one of an object identifier ObjectID, a mesh identifier Mesh ID or a material identifier Material ID.

26. The method according to any one of claims 17 to 25, characterized in that The target operation includes weighted averaging, and performing the target operation on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block includes: The reconstructed image is weighted averaged based on the first rendering intermediate result to obtain a weighted average result, wherein the weighted average result is used to indicate a weighted average pixel value of a virtual reference block to be generated, and the weighted average result is used to generate the virtual reference block.

27. The method according to claim 26, characterized in that The performing weighted averaging on the reconstructed image based on the first rendering intermediate result to obtain a weighted average result includes: A weighted average is performed based on the first rendering intermediate result, the pixel values ​​of the pixels in the reconstructed image, and the image sequence number of the reconstructed image to obtain the weighted average result.

28. The method according to claim 26 or 27, characterized in that The performing weighted averaging on the reconstructed image based on the first rendering intermediate result to obtain a weighted average result includes: A weighted average is performed based on the first rendering intermediate result, the pixel value of the pixel in the reconstructed image, the world coordinates of the pixel in the reconstructed image and the world coordinates of the pixel of the virtual reference block to be generated to obtain the weighted average result.

29. The method according to any one of claims 26 to 28, characterized in that The performing weighted averaging on the reconstructed image based on the first rendering intermediate result to obtain a weighted average result includes: A weighted average is performed based on the first rendering intermediate result, the pixel value of the pixel point in the reconstructed image, the pixel coordinates of the pixel point in the reconstructed image and the pixel coordinates of the pixel point of the virtual reference block to be generated to obtain the weighted average result.

30. The method according to any one of claims 17 to 29, characterized in that The target operation includes hole filling, and the target operation is performed on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block, including: Based on the first rendering intermediate result, holes are filled in the reconstructed image to obtain a hole filling result, wherein the hole filling result is used to indicate the pixel value of the pixel to be filled after the hole is filled, and the pixel to be filled is the pixel that fails the validity check in the virtual reference block to be generated, and the hole filling result is used to generate the virtual reference block.

31. The method according to claim 30, characterized in that The performing hole filling on the reconstructed image based on the first rendering intermediate result to obtain a hole filling result includes: Holes of the reconstructed image are filled based on the first rendering intermediate result and the reprojected pixels of the reconstructed image to obtain the hole filling result.

32. The method according to claim 30 or 31, characterized in that The performing hole filling on the reconstructed image based on the first rendering intermediate result to obtain a hole filling result includes: The reconstructed image is hole-filled based on the first rendering intermediate result and the target pixel point of the reprojected pixel point of the reconstructed image to obtain a hole-filling result, wherein the target pixel point is a pixel point in the reconstructed image whose distance from the reprojected pixel point is less than a first threshold.

33. The method according to any one of claims 30 to 32, characterized in that The performing hole filling on the reconstructed image based on the first rendering intermediate result to obtain a hole filling result includes: Holes are filled in the reconstructed image based on the first rendering intermediate result and non-to-be-filled pixels to obtain the hole filling result, where the non-to-be-filled pixels are pixels that pass the validity check in the virtual reference block to be generated.

34. A coding device, characterized in that: The encoding device includes a rendering module, an encoder and a communication module; The rendering module is used to perform rendering processing on the three-dimensional scene to obtain a rendered image, wherein the rendering intermediate result is generated during the rendering processing, and the rendering intermediate result includes a first rendering intermediate result; The rendering module is further used to perform a target operation on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block, wherein the target operation includes at least one of reprojection, validity check, weighted average, hole filling or intra-frame noise reduction; The encoder is configured to predict the rendered image based on the virtual reference block to obtain a predicted image; The encoder is further configured to encode a residual image between the predicted image and the rendered image to obtain a bit stream; The communication module is used to send the code stream.

35. A decoding device, characterized in that: The decoding device includes a rendering module, a decoder and a communication module; The communication module is used to receive a code stream; The decoder is used to decode the code stream to obtain a residual image of the current image; The rendering module is used to perform rendering processing on the three-dimensional scene to obtain a rendered image, wherein the rendering intermediate result is generated during the rendering processing, and the rendering intermediate result includes a first rendering intermediate result; The rendering module is further used to perform a target operation on the reconstructed image based on the first rendering intermediate result to generate a virtual reference block, wherein the target operation includes at least one of reprojection, validity check, weighted average, hole filling or intra-frame noise reduction; The decoder is further configured to predict the rendered image based on the virtual reference block to obtain a predicted image; The decoder is further used to reconstruct based on the predicted image and the residual image to obtain a reconstructed image of the current frame.

36. A terminal-cloud collaborative system, characterized in that: Includes the encoding device as described in claim 34 and the decoding device as described in claim 35.

37. An encoder, characterized in that Comprising processing circuitry for performing the method according to any one of claims 1 to 16.

38. A decoder, characterized in that Comprising processing circuitry for performing a method according to any one of claims 17 to 33.

39. An encoder, characterized in that include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the encoder to perform the method according to any one of claims 1 to 16.

40. A decoder, characterized in that include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform the method according to any one of claims 17 to 33.

41. A non-transitory computer-readable storage medium, characterized in that The method comprises a program code for executing the method according to any one of claims 1 to 33 when the program code is executed by a computer device.

42. A non-transitory storage medium, characterized in that: The method comprises a code stream according to any one of claims 1 to 33.

43. A computer program product, characterized in that The method comprises a program code for executing the method according to any one of claims 1 to 33 when the program code is executed on a computer or a processor.

Images