Image encoding and decoding method and device

Abstract

The embodiment of the invention provides an image encoding and decoding method and device. The image coding method provided by the embodiment of the invention comprises the following steps: rendering a three-dimensional scene to obtain a to-be-coded image and a rendering intermediate result in a rendering process; preprocessing the rendering intermediate result to obtain an auxiliary image; performing redundancy elimination processing on the to-be-coded image according to the auxiliary image to obtain a redundancy-eliminated image; and encoding the redundancy-removed image to obtain a code stream. According to the embodiment of the invention, the coding and decoding performance can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the image processing technology field, and in particular to an image coding method and device. BACKGROUND

[0002] Many end-cloud collaborative scenarios (for example, cloud gaming, 3D cloud conference, 3D scene collaborative editing, etc.) need to generate a picture through rendering, which is more realistic and can improve the user experience. Rendering requires strong computing power, and the end-side device is limited by objective physical conditions such as device size and power consumption, and the computing power is much weaker than that of the cloud-side server; therefore, rendering is usually deployed on the cloud-side server, the cloud-side server performs rendering, and then the rendered image / video is compressed and sent to the end-side device for display. With the increasing demand for rendering quality and the increasing resolution of display devices, the image quality and resolution of the rendered image / video are also constantly improving, thereby resulting in a considerable bandwidth required for transmitting the rendered image / video.

[0003] Therefore, reducing the transmission bandwidth of the rendered image / video and improving the coding performance are important research directions. SUMMARY

[0004] The present application provides an image coding method and device to improve the coding performance.

[0005] In a first aspect, the present application provides an image coding method, comprising: rendering a three-dimensional scene to obtain an image to be coded and a rendering intermediate result in the rendering process; preprocessing the obtained rendering intermediate result to obtain an auxiliary image; performing de-redundancy processing on the image to be coded according to the obtained auxiliary image to obtain a de-redundancy image; and coding the de-redundancy image to obtain a bitstream.

[0006] In the embodiment of the present application, the rendering intermediate result is preprocessed to obtain an auxiliary image, so that the auxiliary image has a higher similarity with the image to be coded, and more redundant information between the auxiliary image and the image to be coded can be reduced in the de-redundancy processing, the amount of data to be coded is reduced, and the coding performance is improved.

[0007] In one embodiment, the preprocessing includes at least one of the following processing: anti-aliasing processing, feature extraction, or high-pass filtering.

[0008] In one embodiment, the anti-aliasing processing includes one or more of time domain anti-aliasing, multi-sampling anti-aliasing, and blur processing. Further, in one embodiment, the anti-aliasing processing can be consistent with the anti-aliasing processing adopted in the process of rendering the image to be coded.

[0009] In one embodiment, the feature extraction comprises: obtaining a low frequency map according to the rendering intermediate result; and extracting pixel values meeting a preset condition according to the rendering intermediate result and the low frequency map to obtain at least one feature map.

[0010] In one embodiment, the high-pass filtering comprises: filtering the rendering intermediate result to obtain high frequency information in the rendering intermediate result.

[0011] In one embodiment, the filtering the rendering intermediate result to obtain high frequency information in the rendering intermediate result comprises: obtaining a low frequency map according to the rendering intermediate result; and obtaining the high frequency information in the rendering intermediate result by subtracting the low frequency map from the rendering intermediate result.

[0012] In one embodiment, the obtaining the low frequency map according to the rendering intermediate result comprises: dividing the rendering intermediate result to obtain a plurality of to-be-processed tiles; obtaining a plurality of thumbnail maps by averaging pixels in each to-be-processed tile; and obtaining the low frequency map by interpolating pixels in the plurality of thumbnail maps.

[0013] In one embodiment, when the rendering intermediate result comprises a normal map, the preprocessing further comprises performing normal processing on the normal map. In one embodiment, the normal processing comprises: obtaining a light direction and a light intensity of any pixel in the normal map; and obtaining a processed pixel value of the pixel according to a normal direction of the pixel, the light direction and the light intensity. In one embodiment, the normal processing further comprises: multiplying the processed pixel value of the pixel in the normal map with a pixel value at a corresponding position in an albedo map. In one embodiment, the light direction and the light intensity can be encoded into a bitstream.

[0014] In one embodiment, the de-redundancy processing of the to-be-encoded image according to the auxiliary image to obtain a de-redundancy image comprises: obtaining an adjusted auxiliary image, the adjusted auxiliary image being closer to the to-be-encoded image than the auxiliary image; and performing de-redundancy processing according to the adjusted auxiliary image and the to-be-encoded image to obtain the de-redundancy image.

[0015] In one embodiment, the obtaining the adjusted auxiliary image comprises: obtaining at least one set of adjustment parameters according to the auxiliary image and the to-be-encoded image; constructing at least one adjustment function according to the at least one set of adjustment parameters; and adjusting pixels in the corresponding auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image.

[0016] In one embodiment, the obtaining the adjusted auxiliary image comprises: dividing the auxiliary image into a plurality of auxiliary image blocks; for any one auxiliary image block, obtaining at least one set of adjustment parameters according to the auxiliary image block and a corresponding block in the image to be encoded; constructing at least one adjustment function according to the at least one set of adjustment parameters; and adjusting pixels of the corresponding auxiliary image block according to the at least one adjustment function to obtain an adjusted auxiliary image block corresponding to the auxiliary image block.

[0017] In a second aspect, the present application provides an image decoding method, comprising: receiving a code stream; obtaining a de-redundancy image according to the code stream; obtaining a rendering intermediate result; pre-processing the rendering intermediate result to obtain an auxiliary image; and obtaining a reconstructed image according to the auxiliary image and the de-redundancy image.

[0018] In the embodiments of the present application, the rendering intermediate result is pre-processed in a server to obtain an auxiliary image. Since the auxiliary image has a high similarity with the image to be encoded, the de-redundancy image obtained by performing de-redundancy processing on the image to be encoded according to the auxiliary image can reduce more redundant information in the image to be encoded. When such a de-redundancy image is encoded into a code stream and transmitted to a terminal device, the effect of saving the code stream can be achieved. The terminal device can obtain a reconstructed de-redundancy image by decoding the code stream, and obtain an auxiliary image by pre-processing the rendering intermediate result. The reconstructed image can be reconstructed by combining the auxiliary image, and the decoding performance can be improved.

[0019] In one embodiment, the pre-processing comprises at least one of the following: anti-aliasing processing, feature extraction, or high-pass filtering.

[0020] In one embodiment, the anti-aliasing processing comprises one or more of time domain anti-aliasing, multi-sampling anti-aliasing, or blur processing.

[0021] In one embodiment, the feature extraction comprises: obtaining a low-frequency map according to the rendering intermediate result; and extracting pixel values meeting a preset condition according to the rendering intermediate result and the low-frequency map to obtain at least one feature map.

[0022] In one embodiment, the high-pass filtering comprises: filtering the rendering intermediate result to obtain high-frequency information in the rendering intermediate result.

[0023] In one embodiment, the filtering the rendering intermediate result to obtain high-frequency information in the rendering intermediate result comprises: obtaining a low-frequency map according to the rendering intermediate result; and subtracting the low-frequency map from the rendering intermediate result to obtain the high-frequency information.

[0024] In an embodiment, the obtaining the low-frequency map according to the rendering intermediate result comprises: dividing the rendering intermediate result to obtain a plurality of to-be-processed tiles; obtaining a plurality of thumbnails according to the plurality of to-be-processed tiles, the thumbnail being obtained based on a pixel mean value of a corresponding to-be-processed tile; and obtaining the low-frequency map according to the plurality of thumbnails, the low-frequency map being obtained based on an interpolation of pixel values of the plurality of thumbnails.

[0025] In an embodiment, when the rendering intermediate result comprises a normal map, the preprocessing further comprises performing normal processing on the normal map.

[0026] In an embodiment, the normal processing comprises: obtaining a light direction and a light intensity of any one pixel in the normal map; and obtaining a processed pixel value of the pixel according to a normal direction of the any one pixel in the normal map, the light direction and the light intensity.

[0027] In an embodiment, the normal processing further comprises: multiplying the processed pixel value of the any one pixel in the normal map with a pixel value at a corresponding position in an albedo map.

[0028] In an embodiment, the obtaining the light direction and the light intensity of the any one pixel in the normal map comprises: obtaining the light direction and the light intensity of the any one pixel in the normal map from the bitstream.

[0029] In an embodiment, the obtaining the low-frequency map according to the rendering intermediate result comprises: dividing the rendering intermediate result to obtain a plurality of to-be-processed tiles; obtaining a plurality of thumbnails according to the plurality of to-be-processed tiles, the thumbnail being obtained based on a pixel mean value of a corresponding to-be-processed tile; and obtaining the low-frequency map according to the plurality of thumbnails, the low-frequency map being obtained based on an interpolation of pixel values of the plurality of thumbnails.

[0030] In an embodiment, the bitstream further comprises at least one set of adjustment parameters; the obtaining the adjusted auxiliary image comprises: obtaining the at least one set of adjustment parameters from the bitstream; constructing at least one adjustment function according to the at least one set of adjustment parameters; and adjusting pixels of a corresponding auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image.

[0031] In one embodiment, the code stream further comprises at least one set of adjustment parameters; and the obtaining the adjusted auxiliary image comprises: obtaining the at least one set of adjustment parameters from the code stream, any one set of adjustment parameters corresponding to an auxiliary image block, the auxiliary image comprising a plurality of auxiliary image blocks; constructing at least one adjustment function according to the at least one set of adjustment parameters; adjusting pixels of the corresponding auxiliary image block according to the at least one adjustment function to obtain an adjusted auxiliary image block corresponding to the auxiliary image block; and obtaining the adjusted auxiliary image according to the plurality of adjusted auxiliary image blocks.

[0032] In a third aspect, the present application provides an image encoding apparatus, comprising: a preprocessing module configured to render a three-dimensional scene to obtain an image to be encoded and a rendering intermediate result in a rendering process; pre-process the rendering intermediate result to obtain an auxiliary image; a redundancy removal module configured to remove redundancy from the image to be encoded according to the auxiliary image to obtain a redundancy-removed image; and an encoding module configured to encode the redundancy-removed image to obtain a code stream.

[0033] In one embodiment, the preprocessing module is specifically configured to pre-process the rendering intermediate result by at least one of the following: anti-aliasing processing, feature extraction, or high-pass filtering.

[0034] In one embodiment, the anti-aliasing processing comprises one or more of time-domain anti-aliasing, multi-sampling anti-aliasing, and blur processing.

[0035] In one embodiment, the anti-aliasing processing is consistent with anti-aliasing processing adopted in the rendering of the three-dimensional scene to obtain the image to be encoded.

[0036] In one embodiment, the preprocessing module is specifically configured to obtain a low-frequency image from the rendering intermediate result; and extract pixel values meeting a preset condition from the rendering intermediate result and the low-frequency image to obtain at least one feature map.

[0037] In one embodiment, the preprocessing module is specifically configured to filter the rendering intermediate result to obtain high-frequency information in the rendering intermediate result.

[0038] In one embodiment, the preprocessing module is specifically configured to obtain a low-frequency image from the rendering intermediate result; and obtain the high-frequency information by subtracting the low-frequency image from the rendering intermediate result.

[0039] In an embodiment, the pre-processing module is specifically configured to divide the rendering intermediate result to obtain a plurality of to-be-processed tiles; obtain a plurality of thumbnails according to the plurality of to-be-processed tiles, the thumbnail being obtained based on a pixel mean value of a corresponding to-be-processed tile; and obtain the low-frequency map according to the plurality of thumbnails, the low-frequency map being obtained based on an interpolation of pixel values of the plurality of thumbnails.

[0040] In an embodiment, when the rendering intermediate result comprises a normal map, the pre-processing module is further configured to perform normal processing on the normal map.

[0041] In an embodiment, the pre-processing module is specifically configured to obtain a light direction and a light intensity of any one pixel in the normal map; and obtain a processed pixel value of the pixel according to a normal direction of the any one pixel in the normal map, the light direction and the light intensity.

[0042] In an embodiment, the pre-processing module is further configured to multiply the processed pixel value of the any one pixel in the normal map by a pixel value at a corresponding position in an albedo map.

[0043] In an embodiment, the bitstream further comprises the light direction and the light intensity.

[0044] In an embodiment, the redundancy removal module is specifically configured to obtain an adjusted auxiliary image, the adjusted auxiliary image being closer to the to-be-encoded image than the auxiliary image; and perform redundancy removal processing on the adjusted auxiliary image and the to-be-encoded image to obtain the redundancy-removed image.

[0045] In an embodiment, the redundancy removal module is specifically configured to obtain at least one set of adjustment parameters according to the auxiliary image and the to-be-encoded image; construct at least one adjustment function according to the at least one set of adjustment parameters; and adjust pixels of a corresponding auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image.

[0046] In an embodiment, the redundancy removal module is specifically configured to divide the auxiliary image into a plurality of auxiliary image blocks; for any one auxiliary image block, obtain at least one set of adjustment parameters according to the auxiliary image block and a corresponding block in the to-be-encoded image; construct at least one adjustment function according to the at least one set of adjustment parameters; and adjust pixels of a corresponding auxiliary image block according to the at least one adjustment function to obtain an adjusted auxiliary image block corresponding to the auxiliary image block.

[0047] In a fourth aspect, the present application provides an image decoding apparatus, comprising: a receiving module configured to receive a code stream; a decoding module configured to obtain a de-redundancy image according to the code stream; a preprocessing module configured to obtain a rendering intermediate result; pre-process the rendering intermediate result to obtain an auxiliary image; and a reconstruction module configured to obtain a reconstructed image according to the auxiliary image and the de-redundancy image.

[0048] In one embodiment, the pre-processing performed by the preprocessing module on the rendering intermediate result comprises at least one of the following: anti-aliasing processing, feature extraction, or high-pass filtering.

[0049] In one embodiment, the anti-aliasing processing comprises one or more of time-domain anti-aliasing, multi-sampling anti-aliasing, or blur processing.

[0050] In one embodiment, the preprocessing module is specifically configured to obtain a low-frequency map according to the rendering intermediate result; and extract pixel values that meet a preset condition from the rendering intermediate result and the low-frequency map to obtain at least one feature map.

[0051] In one embodiment, the preprocessing module is specifically configured to filter the rendering intermediate result to obtain high-frequency information in the rendering intermediate result.

[0052] In one embodiment, the preprocessing module is specifically configured to obtain a low-frequency map according to the rendering intermediate result; and obtain the high-frequency information by subtracting the low-frequency map from the rendering intermediate result.

[0053] In one embodiment, the preprocessing module is specifically configured to divide the rendering intermediate result into a plurality of to-be-processed tiles; obtain a plurality of thumbnails according to the plurality of to-be-processed tiles, wherein the thumbnail is obtained based on the average pixel value of the corresponding to-be-processed tile; and obtain the low-frequency map based on the pixel values of the plurality of thumbnails by interpolation.

[0054] In one embodiment, when the rendering intermediate result comprises a normal map, the preprocessing module is further configured to perform normal processing on the normal map.

[0055] In one embodiment, the preprocessing module is specifically configured to obtain a light direction and a light intensity of any one pixel in the normal map; and obtain a processed pixel value of the pixel according to the normal direction of the pixel, the light direction, and the light intensity.

[0056] In one embodiment, the preprocessing module is further configured to multiply the processed pixel value of any one pixel in the normal map by the pixel value at the corresponding position in an albedo map.

[0057] In an embodiment, the preprocessing module is specifically configured to obtain the light direction and the light intensity of any pixel in the normal map from the bitstream.

[0058] In an embodiment, the reconstruction module is specifically configured to obtain an adjusted auxiliary image, the adjusted auxiliary image being closer to the reconstructed image than the auxiliary image; and reconstruct according to the adjusted auxiliary image and the de-redundancy image to obtain the reconstructed image.

[0059] In an embodiment, the bitstream further comprises at least one set of adjustment parameters; and the reconstruction module is specifically configured to obtain the at least one set of adjustment parameters from the bitstream; construct at least one adjustment function according to the at least one set of adjustment parameters; and adjust the pixels of the corresponding auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image.

[0060] In an embodiment, the bitstream further comprises at least one set of adjustment parameters; and the reconstruction module is specifically configured to obtain the at least one set of adjustment parameters from the bitstream, any set of adjustment parameters corresponding to an auxiliary image block, the auxiliary image comprising a plurality of auxiliary image blocks; construct at least one adjustment function according to the at least one set of adjustment parameters; adjust the pixels of the corresponding auxiliary image block according to the at least one adjustment function to obtain an adjusted auxiliary image block corresponding to the auxiliary image block; and obtain the adjusted auxiliary image according to a plurality of the adjusted auxiliary image blocks.

[0061] In a fifth aspect, the present application provides an end-cloud collaborative system, comprising: a server and a terminal device; wherein the server is configured to perform rendering processing on a three-dimensional scene to obtain an image to be encoded and a rendering intermediate result in a rendering process; perform preprocessing on the rendering intermediate result to obtain a first auxiliary image; perform de-redundancy processing on the image to be encoded based on the first auxiliary image to obtain a de-redundancy image; encode the de-redundancy image to obtain a bitstream; and transmit the bitstream.

[0062] The terminal device is configured to receive the bitstream; decode the bitstream to obtain a decoding result, the decoding result comprising the de-redundancy image; obtain a second auxiliary image; and reconstruct based on the de-redundancy image and the second auxiliary image to obtain a reconstructed image.

[0063] In an embodiment, the resolution of the first auxiliary image is different from the resolution of the second auxiliary image; and / or the content of the first auxiliary image is different from the content of the second auxiliary image.

[0064] In an embodiment, the resolution of the second auxiliary image is higher than the resolution of the first auxiliary image.

[0065] In a sixth aspect, the present application provides an apparatus, comprising: one or more processors; a memory storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of the first to second aspects.

[0066] In a seventh aspect, the present application provides a computer-readable storage medium, comprising a computer program, when the computer program is executed on an apparatus, the apparatus executes the method according to any one of the first to second aspects.

[0067] In an eighth aspect, the present application provides a computer program, when the computer program is executed on an apparatus, the apparatus executes the method according to any one of the first to second aspects.

[0068] In a ninth aspect, the present application provides a computer program product, the computer program product comprises computer program code, when the computer program code is executed on an apparatus, the apparatus executes the method according to any one of the first to second aspects.

[0069] In a tenth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a video bitstream obtained by executing the method according to any one of the first aspect on one or more processors.

[0070] In an eleventh aspect, the present application provides a method for transmitting an encoded bitstream of video data, the method comprising: obtaining a bitstream from a storage medium, the bitstream being generated according to the method of any one of the first aspect and stored in the storage medium; and transmitting the bitstream.

[0071] In a twelfth aspect, the present application provides a system for transmitting an encoded bitstream of video data, the system comprising: an obtaining unit configured to obtain a bitstream from a storage medium, the bitstream being generated according to the method of any one of the first aspect and stored in the storage medium; and a transmitting unit configured to transmit the bitstream.

[0072] In a thirteenth aspect, the present application provides a method for storing an encoded bitstream of video data, the method comprising: receiving the bitstream generated according to the method of any one of the first aspect; and storing the bitstream into a storage medium.

[0073] In a fourteenth aspect, the present application provides a system for storing an encoded bitstream of video data, comprising: a receiving unit configured to receive the bitstream generated according to the method of any one of the first aspect; and a storing unit configured to store the bitstream. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1a An exemplary schematic diagram of an application scenario of an embodiment of the present application;

[0075] Figure 1b An exemplary schematic diagram of an end-cloud collaborative system of an embodiment of the present application;

[0076] Figure 2a An exemplary schematic diagram of an encoding process of an embodiment of the present application;

[0077] Figure 2b An exemplary schematic diagram of an encoding framework of an embodiment of the present application;

[0078] Figure 3a An exemplary schematic diagram of a decoding process of an embodiment of the present application;

[0079] Figure 3b An exemplary schematic diagram of a decoding framework;

[0080] Figure 4 A flowchart of a process 400 of an image encoding method provided by an embodiment of the present application;

[0081] Figure 5 An exemplary schematic diagram of a geometric partitioning mode;

[0082] Figure 6 An exemplary schematic diagram of an Albedo map without using anti-aliasing processing and an image to be encoded using anti-aliasing processing of an embodiment of the present application;

[0083] Figure 7 An exemplary legend in feature extraction of an embodiment of the present application;

[0084] Figure 8 A flowchart of a normal processing of an embodiment of the present application;

[0085] Figure 9 A flowchart of a process 900 of an image decoding method provided by an embodiment of the present application;

[0086] Figure 10 A flowchart of an encoding end provided in an embodiment of the present application;

[0087] Figure 11 A flowchart of a decoding end of an embodiment of the present application;

[0088] Figure 12 An exemplary schematic diagram of an end-cloud encoding and decoding architecture of an embodiment of the present application;

[0089] Figure 13 An exemplary comparison diagram of a rendering result of an embodiment of the present application;

[0090] Figure 14 A schematic diagram of an end-to-cloud coding architecture according to an embodiment of the present application;

[0091] Figure 15 A schematic diagram of an end-to-cloud coding architecture according to an embodiment of the present application;

[0092] Figure 16 A schematic diagram of an image encoding apparatus 1600 according to an embodiment of the present application;

[0093] Figure 17 A schematic diagram of an image decoding apparatus 1700 according to an embodiment of the present application;

[0094] Figure 18 A schematic block diagram of an apparatus according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0095] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0096] The terms "first", "second", and the like in the description and claims of the present application are intended to distinguish between similar objects but are not intended to indicate or imply a relative importance between the objects and are not intended to indicate or imply a serial order, unless otherwise specified. In addition, the terms "comprise", "comprising", "have", "having", and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, product, or apparatus that comprises a list of steps or units. The process, system, product, or apparatus is not necessarily limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products, or apparatuses.

[0097] It should be understood that in the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0098] The following are the key words involved in the embodiments of the present application:

[0099] Rendered intermediate result: The intermediate data generated by a graphics rendering engine in the process of generating a to-be-displayed image / video for generating the to-be-displayed image / video. The intermediate data can include, but is not limited to, one or more of an albedo map, a normal map, a low-quality rendering result, a position map, a specular intensity map, a depth map, an ambient occlusion map, a computer graphic motion vector (CGMV), a Mesh ID, a Material ID, and the like.

[0100] Rendering parameter: The input of a graphics rendering engine, including various parameters for rendering, the position and color vector of all light sources, the position vector of a player or observer, the sampling method of each texture, and the position coordinates of each scene object, etc. Motion trajectory of moving objects, bone animation parameters.

[0101] Reference frame: A concept in video encoding methods, generally similar to the image to be encoded. The image generated by subtracting the reference frame from the image to be encoded will have a lower information content and be more conducive to encoding.

[0102] Auxiliary image: In the present application, further preprocessing of the rendered intermediate result image will be involved, and the resulting image is an auxiliary image. The auxiliary image often has similar visual features to the final rendering result but cannot directly serve as a reference frame. The present application uses these auxiliary images to remove the redundant information of the image to be encoded. The aforementioned preprocessing can include any one or more of anti-aliasing processing, feature extraction, normal processing, or high-pass filtering.

[0103] Adjustment: the output value of a function after inputting each pixel in the auxiliary image into the function respectively is taken as the pixel value of the new image, and the adjusted image can be subtracted from the to-be-encoded image like the reference frame to generate a de-redundancy image with lower information quantity.

[0104] Adjustment parameter: a parameter used to describe the specific operation of adjustment, which is the coefficient of the function used for adjustment.

[0105] Standard video encoder and standard video decoder: can be implemented according to the industry video compression standard, for example, the standard can be ITU-T H.264, H.265 / HEVC, H.266 / VVC, AVS2, AVS3 standard, or an extension of such a standard. The technology of the present application is not limited to any specific coding standard.

[0106] Figure 1a An exemplary schematic diagram of an application scenario of an embodiment of the present application. As shown in the figure, the embodiment of the present application can be applied to various end-cloud collaboration scenarios, for example, cloud gaming, 3D cloud conference, 3D scene collaborative editing and production, etc., which are not specifically limited. The embodiment of the present application is described taking cloud gaming as an example. Figure 1a

[0107] Reference is made to Figure 1a ​In an embodiment, the terminal device can include a mobile phone, a personal computer (PC), virtual reality (VR) glasses, an augmented reality (AR) helmet, a tablet computer, and other terminal devices. The first rendering parameter is sent to a center server (e.g., a cloud game server) through an edge server. Then, the center server can perform rendering processing on the three-dimensional scene based on the rendering parameter (the rendering parameter can include only the first rendering parameter uploaded by the terminal device, or include the first rendering parameter uploaded by the terminal device and a second rendering parameter generated by the center server) to obtain a rendered image (e.g., a game image). Subsequently, the center server performs preprocessing on the rendering intermediate result generated in the rendering processing to obtain an auxiliary image, performs de-redundancy processing on the rendered image based on the auxiliary image to obtain a de-redundancy image, encodes the de-redundancy image to obtain a code stream, and distributes the code stream to each terminal device through the edge server. Subsequently, the terminal device can perform rendering processing on the three-dimensional scene based on the rendering parameter (the rendering parameter includes the first rendering parameter and a fifth rendering parameter, the fifth rendering parameter is generated by the terminal device, and the fifth rendering parameter can include the same parameters as the second rendering parameter), and generate a rendering intermediate result in the rendering processing. The rendering intermediate result is preprocessed to obtain an auxiliary image. The code stream is decoded to obtain a de-redundancy image. Then, the terminal device reconstructs based on the auxiliary image and the de-redundancy image to obtain a reconstructed image.

[0108] It should be understood that the image to be encoded and the rendered image described in the present application are consistent in meaning, i.e., the image to be encoded is obtained by rendering the three-dimensional scene by the server / cloud / encoding end.

[0109] Reference Figure 1aIn an embodiment, the terminal device can include a mobile phone, a PC, a VR glasses, an AR helmet, a tablet computer and other terminal devices, and the first rendering parameter is sent to the center server through the edge server. Then, the center server can perform rendering processing on the three-dimensional scene based on the rendering parameter (the rendering parameter can include only the first rendering parameter uploaded by the terminal device, or include the first rendering parameter uploaded by the terminal device and the second rendering parameter generated by the center server) to obtain a rendered image. Subsequently, the center server performs preprocessing on the rendering intermediate result generated in the rendering processing to obtain an auxiliary image, adjusts the auxiliary image to obtain an adjusted auxiliary image, and then performs de-redundancy processing on the rendered image based on the adjusted auxiliary image to obtain a de-redundancy image, and encodes the de-redundancy image; and encodes the adjustment parameter or the adjustment function used in the foregoing adjustment processing to obtain a code stream and distribute the code stream to each terminal device through the edge server. Subsequently, the terminal device can perform rendering processing on the three-dimensional scene based on the rendering parameter (the rendering parameter includes the first rendering parameter and the fifth rendering parameter), and generate a rendering intermediate result in the rendering processing; and perform preprocessing on the rendering intermediate result to obtain an auxiliary image. At the same time, the code stream is decoded to obtain the de-redundancy image and the adjustment parameter or the adjustment function used in the subsequent adjustment processing. Then, the terminal device adjusts the auxiliary image to obtain an adjusted auxiliary image, and then reconstructs based on the adjusted auxiliary image and the de-redundancy image to obtain a reconstructed image.

[0110] The embodiment of the present application can obtain an auxiliary image with higher similarity to the rendered image by preprocessing the rendering intermediate result, and then perform de-redundancy processing on the rendered image based on the auxiliary image, which can better reduce the more redundant information of the rendered image and improve the encoding performance and the size of the code stream.

[0111] For example, the rendering parameter can refer to all parameters input to the graphics rendering engine and required by the graphics rendering engine for rendering processing, which can include various rendering parameters, positions and color vectors of all light sources, position vectors of players or observers, sampling methods of each texture, position coordinates of each scene object, motion trajectories of moving objects, bone animation parameters, and the like, which are not limited in the embodiment of the present application. The first rendering parameter and the second rendering parameter can constitute the rendering parameter (i.e., all parameters input to the graphics rendering engine and required by the graphics rendering engine for rendering).

[0112] Based on the application scenario of Figure 1a , the embodiment of the present application proposes an end-cloud collaborative system, which can be as shown in Figure 1b .

[0113] Figure 1b is an exemplary schematic diagram of the end-cloud collaborative system of the embodiment of the present application. AsFigure 1b As shown, the end-cloud collaborative system can include a server and a terminal device.

[0114] Referring to Figure 1b For example, the server can refer to Figure 1a a central server in a cloud computing center. The server can be a single server or a server cluster, and the embodiments of the present application do not make specific limitations on this.

[0115] For example, the server can include a first rendering module, an encoder, and a first communication module. It should be understood that Figure 1b This is only one example of the embodiments of the present application. The server of the embodiments of the present application can include more or fewer modules than Figure 1b shown, and the embodiments of the present application do not make specific limitations on this.

[0116] For example, the first rendering module can be used for rendering processing of a three-dimensional scene to obtain a rendered image and a rendering intermediate result.

[0117] For example, the encoder can be used for pre-processing the rendering intermediate result to obtain an auxiliary image, and performing de-redundancy processing on the rendered image based on the auxiliary image to obtain a de-redundancy image.

[0118] For example, the encoder can be used for encoding the de-redundancy image to obtain a code stream.

[0119] For example, the first communication module can be used for communication with other electronic devices, for example, the code stream output by the encoder can be sent to the terminal device.

[0120] Referring to Figure 1b For example, the terminal device includes but is not limited to: a personal computer, a computer workstation, a smart phone, a tablet computer, a server, a smart camera, a smart car or other types of cellular phones, a media consumption device, a wearable device (such as a VR / AR headset, VR glasses), a set-top box, a game console, etc.

[0121] For example, the terminal device can include a second communication module, a second rendering module, a decoder, and a display module. It should be understood that Figure 1b This is only one example of the embodiments of the present application. The terminal device of the embodiments of the present application can include more or fewer modules than Figure 1b shown, and the embodiments of the present application do not make specific limitations on this.

[0122] For example, the second communication module can be used for communication with other electronic devices, for example, receiving the code stream sent by the server.

[0123] Exemplarily, the second rendering module can be configured to perform rendering processing on the three-dimensional scene to obtain a rendering intermediate result. It should be understood that the second rendering module can also generate a rendered image.

[0124] Exemplarily, the decoder can be configured to perform pre-processing on the rendering intermediate result to obtain an auxiliary image.

[0125] Exemplarily, the decoder can be configured to decode the bitstream to obtain a de-redundancy image, and to obtain a reconstructed image based on the auxiliary image and the de-redundancy image.

[0126] Exemplarily, the display module can be configured to display the reconstructed image.

[0127] It should be understood that the video coding standards used by the encoder and the decoder are not limited by the embodiments of the present application, for example, can include but are not limited to: H.264 / Advanced Video Coding (AVC), H.265 / High Efficiency Video Coding (HEVC), H.266 / Versatile Video Coding (VVC), AV1 (AOMedia Video 1, "AOMedia" refers to a video coding standard developed by the Open Media Video Alliance), and the like, and the extension standards of these video coding standards; in addition, it can also include new video coding standards and extension standards generated with the development of video coding technology.

[0128] The following is based on Figure 1a and Figure 1b , the coding process is described.

[0129] Figure 2a is an exemplary schematic diagram of the coding process of the embodiments of the present application. The embodiments of the present application take the coding of a frame of rendered image as an example for description.

[0130] S201, the server performs rendering processing on the three-dimensional scene to obtain a rendered image and a rendering intermediate result.

[0131] Exemplarily, the rendering processing process of the graphics rendering engine can be as follows: the image engine can perform geometric transformation, projection transformation, perspective transformation and window clipping on the three-dimensional object model in the three-dimensional scene based on the rendering parameters, and then generate a rendered image through the obtained material and light information.

[0132] Exemplarily, the graphic rendering engine in the first rendering module generates a rendering intermediate result in a rendering process (the rendering process refers to a time period between a time point when the rendering parameter is obtained from the graphic rendering engine and a time point when the rendered image is obtained).

[0133] Optionally, in an embodiment, a first rendering intermediate result can be selected from the rendering intermediate results. Exemplarily, a part of the rendering intermediate results can be selected as the first rendering intermediate result.

[0134] The rendering parameter used in the rendering process can include one or more of a rendering parameter obtained from the terminal device, a locally stored rendering parameter, or a pre-set rendering parameter.

[0135] Exemplarily, the server can obtain the rendering parameter; then, the rendering parameter can be input to the graphic rendering engine, and the graphic rendering engine can perform rendering processing on the three-dimensional scene based on the rendering parameter to obtain the rendered image. The graphic rendering engine can exemplarily be the first rendering module in the server. Figure 1b

[0136] Exemplarily, the server can obtain the first rendering parameter from the terminal device (wherein the first rendering parameter is generated by the terminal device). In an embodiment, when the first rendering parameter generated by the terminal device is all the parameters required by the graphic rendering engine for rendering processing, the server can determine the first rendering parameter as the rendering parameter; that is, the rendering parameter obtained by the server can only include the first rendering parameter. In an embodiment, when the first rendering parameter generated by the terminal device is part of the parameters required by the graphic rendering engine for rendering processing, the server can further generate a second rendering parameter; then, the first rendering parameter and the second rendering parameter are determined as the rendering parameter. That is, the rendering parameter obtained by the server can include the first rendering parameter and the second rendering parameter. The first rendering parameter and the second rendering parameter are described in detail in subsequent embodiments.

[0137] It should be noted that the server can obtain the first rendering parameter from the terminal device according to a preset period, wherein the preset period can be set according to requirements, and the embodiments of the present application do not make specific limitations thereon. It should be understood that the first rendering parameter obtained by the server from the terminal device each time can be used to render one or more frames of images.

[0138] It should be noted that S201 can be performed by the first rendering module in the server, and the embodiments of the present application do not make specific limitations thereon. S202-S204 described below can be performed by the encoder in the server. Figure 1b Figure 1b

[0139] ​​​S202, the server pre-processes the rendering intermediate result to obtain an auxiliary image.

[0140] The rendering intermediate result or the first rendering intermediate result is pre-processed to obtain an auxiliary image. In an embodiment, the rendering intermediate result, a partial image of the rendering intermediate result or the first rendering intermediate result can be pre-processed to obtain the auxiliary image, and the pre-processing in this embodiment can include one or more of, but not limited to, anti-aliasing processing, feature extraction, high-pass filtering, normal processing, etc.

[0141] S203, the server de-redundantizes the rendered image based on the auxiliary image to obtain a de-redundant image.

[0142] S204, the server encodes the de-redundant image and writes the encoding data of the de-redundant image into a bitstream.

[0143] Figure 2b An exemplary schematic diagram of an encoding framework of an embodiment of the present application. Figure 2b An embodiment of the present application shows Figure 1b An encoding framework of an encoder in the present application. Figure 2b The to-be-encoded video in the present application is a de-redundant video, which can include multiple de-redundant images.

[0144] Referring to Figure 2b , the encoder can include, for example, a blocking module, an intra prediction module, an inter prediction module, a transform module, a quantization module, an entropy encoding module, a dequantization module, an inverse transform module, a loop filtering module and a memory.

[0145] The encoding process of the encoder can be as follows: after the blocking module of the encoder obtains the to-be-encoded video, it can block each de-redundant image in the to-be-encoded video to obtain a to-be-encoded block. For each to-be-encoded block, the decision module (not shown in the present application) can determine whether to perform inter prediction or intra prediction on the to-be-encoded block. Figure 2b

[0146] When it is determined to perform intra prediction, the intra prediction module can perform intra prediction to determine a prediction block corresponding to the to-be-encoded block. After the prediction block is obtained, a residual block between the to-be-encoded block and the prediction block can be determined; then, the residual block is input into the transform module, and the transform module transforms the residual block to obtain a transform result and outputs the transform result to the quantization module. Subsequently, the quantization module can quantize the transform result to obtain a quantization result and output the quantization result to the entropy encoding module; then, the entropy encoding module can entropy encode the quantization result to obtain encoding data of the residual block, and write the encoding data of the residual block into a bitstream.

[0147] ​For example, the quantization module can further output the quantization result to the inverse quantization module, and the inverse quantization module performs inverse quantization on the quantization result to obtain an inverse quantization result and output the inverse quantization result to the inverse transform module. Then, the inverse transform module performs inverse transform on the inverse quantization result to obtain a decoded residual block. Then, the decoded residual block can be superimposed with the prediction block output by the intra prediction module to obtain a reconstructed block. Subsequently, the reconstructed block is input to the loop filter module, and the loop filter module performs loop filtering on the reconstructed block to obtain a filtered reconstructed block and output the filtered reconstructed block to the memory. (Wherein, after obtaining the filtered reconstructed block, the loop filter module can splice the filtered reconstructed block to the corresponding position of the to-be-reconstructed image; after obtaining a reconstructed image, the reconstructed image can be output to the memory for storage.)

[0148] When it is determined to be inter prediction, inter prediction can be performed by the inter prediction module to determine a prediction block corresponding to the to-be-encoded block.

[0149] In an embodiment, the inter prediction module can select a reference frame from the reference frame list of the memory as a reference frame of the de-redundancy image. For the to-be-encoded block in the de-redundancy image, a matching prediction block can be found in the reference frame.

[0150] In an embodiment, the inter prediction module can select an original reference frame from the reference frame list of the memory as a candidate reference frame of the de-redundancy image; wherein the original reference frame refers to a reconstructed image. For the to-be-encoded block in the de-redundancy image, inter prediction can be performed based on multiple candidate reference frames to determine multiple prediction blocks; wherein one candidate reference frame corresponds to one prediction block; then one optimal prediction block can be selected from the multiple prediction blocks (for example, the prediction block with the minimum rate-distortion cost can be determined as the optimal prediction block as an evaluation standard).

[0151] After obtaining the prediction block, a residual block between the to-be-encoded block and the prediction block can be determined; then the residual block is input to the transform module, and the transform module performs transform on the residual block to obtain a transform result and output the transform result to the quantization module. Subsequently, the quantization module can quantize the transform result to obtain a quantization result and output the quantization result to the entropy encoding module; then the entropy encoding module can entropy encode the quantization result to obtain the encoding data of the residual block, and write the encoding data of the residual block into the code stream (that is, the code stream obtained in S204 described above).

[0152] The quantization module can output the quantization result to the inverse quantization module, the inverse quantization module can perform inverse quantization on the quantization result to obtain an inverse quantization result, and the inverse quantization module can output the inverse quantization result to the inverse transform module. Then, the inverse transform module can perform inverse transform on the inverse quantization result to obtain a decoded residual block. Then, the decoded residual block can be superimposed with the prediction block output by the intra prediction module to obtain a reconstructed block. Subsequently, the reconstructed block can be input to the loop filter module, the loop filter module can perform loop filtering on the reconstructed block to obtain a filtered reconstructed block, and the loop filter module can output the filtered reconstructed block to the memory. (After obtaining the filtered reconstructed block, the loop filter module can splice the filtered reconstructed block to a corresponding position of the to-be-reconstructed image. After obtaining a reconstructed image, the reconstructed image can be output to the memory for storage.)

[0153] Figure 3a An example schematic diagram of a decoding process of an embodiment of the present application is shown. Figure 3a An example decoding process is shown. Figure 2a An example decoding process is shown. Figure 3a An embodiment is described by taking decoding of an image as an example.

[0154] S301, the terminal device receives a code stream.

[0155] After the server transmits the code stream, the terminal device can receive the code stream.

[0156] S302, the terminal device decodes the code stream to obtain a decoding result, the decoding result including a de-redundancy image.

[0157] Figure 3b An example decoding framework schematic diagram is shown. Figure 3b An embodiment of the present application shows Figure 1b A decoding framework of a decoder in an embodiment.

[0158] Referring to Figure 3b , the decoder can include an entropy decoding module, an inverse quantization module, an inverse transform module, an intra prediction module, an inter prediction module, a loop filter module, and a memory.

[0159] After receiving the code stream, the terminal device can decode the code stream to obtain a decoding result. The decoding process can be as follows: the coding data of the residual block is extracted from the code stream, the coding data of the residual block is input to the entropy decoding module, the entropy decoding module performs entropy decoding on the coding data of the residual block to obtain entropy decoding data. Then, the entropy decoding data can be input to the inverse quantization module, the inverse quantization module can perform inverse quantization on the entropy decoding data to obtain inverse quantization data, and the inverse quantization data can be input to the inverse transform module. Subsequently, the inverse transform module performs inverse transform on the inverse quantization data to obtain a decoded residual block.

[0160] It should be understood that when the residual block is lossy coded at the encoding end, the residual block decoded at the decoding end is different from the residual block coded at the encoding end. When the residual block is lossless coded at the encoding end, the residual block decoded at the decoding end is the same as the residual block coded at the encoding end.

[0161] It should be noted that the code stream description information used to describe the code stream can also be extracted from the code stream, at which time, the code stream description information does not need to be subjected to entropy decoding, inverse quantization, and inverse transformation, and the like. The code stream description information can be used in the subsequent decoding process.

[0162] For example, when it is determined according to the code stream description information decoded from the code stream that it is intra prediction, intra prediction can be performed by the intra prediction module to obtain a prediction block. Subsequently, the decoded residual block and the prediction block can be superimposed to obtain a reconstructed block. Then, the reconstructed block can be subjected to loop filtering to obtain a filtered reconstructed block and output the filtered reconstructed block to the memory. (Wherein, the loop filtering module can splice the filtered reconstructed block to the corresponding position of the to-be-reconstructed image after obtaining the filtered reconstructed block; after obtaining a reconstructed image, the reconstructed image can be output to the memory for storage.)

[0163] For example, when it is determined according to the code stream description information decoded from the code stream that it is inter prediction, inter prediction can be performed by the inter prediction module to obtain a prediction block.

[0164] In an embodiment, when the inter prediction module determines from the code stream description information decoded from the code stream that the reference frame is an original reference frame, the inter prediction module can select the corresponding original reference frame from the reference frame list of the memory as the reference frame; then, the prediction block can be determined based on the original reference frame.

[0165] After obtaining the prediction block, the decoded residual block and the prediction block can be superimposed to obtain a reconstructed block. Then, the reconstructed block can be subjected to loop filtering to obtain a filtered reconstructed block and output the filtered reconstructed block to the memory. (Wherein, the loop filtering module can splice the filtered reconstructed block to the corresponding position of the to-be-reconstructed image after obtaining the filtered reconstructed block; after obtaining a reconstructed image, the reconstructed image can be output to the memory.) Figure 3b The reconstructed image is a de-redundancy image, and a plurality of de-redundancy images form a de-redundancy video.

[0166] It should be understood that the terminal device can also generate the code stream based on the first rendering parameter, and then send the code stream to the server according to a preset period. In this way, the server can obtain the first rendering parameter. In addition, since the data amount of the first rendering parameter is small, the terminal device can encode the first rendering parameter and write the encoded data of the first rendering parameter into the code stream, and then send the code stream to the server, or the terminal device can directly write the first rendering parameter into the code stream without encoding the first rendering parameter, and then send the code stream to the server. The embodiments of the present application do not make specific limitations in this regard.

[0167] In S303, the terminal device performs rendering processing on the three-dimensional scene to obtain a rendering intermediate result generated in the rendering processing.

[0168] For example, the second rendering module (which can be a graphic rendering engine included in the second rendering module) can perform rendering processing on the three-dimensional scene.

[0169] For example, the second rendering module can perform rendering processing on the three-dimensional scene based on the rendering parameter. When the first rendering parameter uploaded by the terminal device to the server is all parameters required for the graphic rendering engine to perform rendering processing, the second rendering module can perform rendering processing on the three-dimensional scene based on the first rendering parameter corresponding to the current frame. When the first rendering parameter uploaded by the terminal device is part of the parameters required for the graphic rendering engine to perform rendering processing, the terminal device can further generate a fifth rendering parameter, and then the second rendering module can perform rendering processing on the three-dimensional scene based on the first rendering parameter corresponding to the current frame and the fifth rendering parameter. It should be noted that the type of the parameter included in the fifth rendering parameter generated by the terminal device is the same as the type of the parameter included in the second rendering parameter generated by the central server, and the precision of the parameter included in the fifth rendering parameter is less than or equal to the precision of the parameter included in the second rendering parameter.

[0170] Optionally, in an embodiment, a first rendering intermediate result can be selected from the rendering intermediate results. For example, part of the rendering intermediate results can be selected as the first rendering intermediate result.

[0171] It should be noted that the type of the intermediate result included in the first rendering intermediate result generated by the terminal device is the same as the type of the intermediate result included in the first rendering intermediate result generated by the server, and the precision of the intermediate result included in the first rendering intermediate result generated by the terminal device can be the same as or different from the precision of the intermediate result included in the first rendering intermediate result generated by the server, for example, the former is higher than or equal to the latter. In addition, the texture of the intermediate result included in the first rendering intermediate result generated by the terminal device can be the same as or different from the texture of the intermediate result included in the first rendering intermediate result generated by the server, and no specific limitation is made in this regard.

[0172] It should be noted that S303 can be performed by the second rendering module in the server, and the embodiments of the present application do not make specific limitations on this. S302, S304 and S305 can be performed by a decoder in the terminal device. Figure 1b

[0173] S304, the terminal device pre-processes the rendering intermediate result to obtain an auxiliary image.

[0174] The rendering intermediate result or the first rendering intermediate result is pre-processed to obtain the auxiliary image. In an embodiment, the rendering intermediate result, a partial image of the rendering intermediate result or the first rendering intermediate result can be pre-processed to obtain the auxiliary image. The pre-processing in this embodiment can include, but is not limited to, one or more of anti-aliasing processing, feature extraction, high-pass filtering, normal processing, etc.

[0175] S305, the terminal device reconstructs based on the auxiliary image and the de-redundancy image to obtain a reconstructed image.

[0176] Since the auxiliary image is obtained by pre-processing the rendering intermediate result, the auxiliary image can be closer to the auxiliary image on the encoding side. The reconstruction based on the auxiliary image and the de-redundancy image can ensure the image reconstruction quality.

[0177] Figure 4 A flowchart of a process 400 of the image encoding method provided by the embodiments of the present application is shown. The process 400 can be performed by the server described above. The process 400 is described as a series of steps or operations. It should be understood that the process 400 can be performed in various sequences and / or simultaneously, and is not limited to the execution sequence shown. Figure 4 The process 400 can include the following steps:

[0178] Step 401, rendering a three-dimensional scene to obtain an image to be encoded and a rendering intermediate result in the rendering process.

[0179] The server can process each image frame in sequence according to the sequence of image sequences in the video, and the image to be encoded can refer to the image frame currently being processed by the server.

[0180] The server renders a three-dimensional scene to obtain an image to be encoded and a rendering intermediate result in the rendering process. The rendering processing manner of this step can refer to the description of step S201 above, or refer to the existing rendering technology.

[0181] The image to be encoded and the rendering intermediate result can refer to an entire image, or a partial image of a frame of image, for example, an image block, a region of interest (ROI), etc. The size and shape of the partial image are not specifically limited. ​

[0182] Step 402, pre-processing the rendering intermediate result to obtain an auxiliary image.

[0183] In the embodiments of the present application, the server can pre-process the rendering intermediate result, and then obtain one or more auxiliary images for subsequent processing. The aforementioned pre-processing can include at least one of the following: anti-aliasing processing, feature extraction, or high-pass filtering.

[0184] i. Anti-aliasing processing

[0185] When rendering an object, the graphics rendering engine may, due to the resolution, produce unnatural sawtooth edges on the object, affecting the visual effect. In order to reduce the generation of sawtooth phenomenon, when generating the image to be encoded, the graphics rendering engine will use anti-aliasing processing. Typical anti-aliasing processing includes temporal anti-aliasing (TAA), multisample anti-aliasing (MSAA), blur processing, etc.

[0186] The anti-aliasing processing can improve the visual effect of the image to be encoded, but it will also cause differences in features between the image to be encoded and the rendering intermediate result (e.g., the Albedo map), such as Figure 6 ( Figure 6 As shown in the comparison diagram of the Albedo map not using anti-aliasing processing and the image to be encoded using anti-aliasing processing of the embodiments of the present application, the use of anti-aliasing processing causes the Albedo map and the image to be encoded to have obvious differences in vision, which may affect the effect of subsequent redundancy removal processing. Therefore, in order to eliminate such obvious differences in vision and ensure that the auxiliary image has higher similarity with the image to be encoded, the server can use at least one of the following methods:

[0187] 1) Anti-aliasing processing is performed on the rendering intermediate result, which is consistent with the anti-aliasing processing used in the process of rendering the three-dimensional scene to obtain the image to be encoded. The graphics rendering engine renders the three-dimensional scene, first outputs the rendering intermediate result, and finally outputs the image to be encoded. One or more of the following processing methods can be used after the rendering intermediate result is output: TAA, MSAA, blur, etc. Therefore, the image to be encoded is anti-aliased, and the rendering intermediate result is not anti-aliased. The rendering intermediate result is anti-aliased in the same way as the image to be encoded, which can improve the visual effect of the rendering intermediate result and reduce the differences in features between the image to be encoded and the rendering intermediate result.

[0188] 2) one or more of TAA, MSAA, blur, and other anti-aliasing processing methods are performed on the rendering intermediate result to reduce the sharpness of the edge aliasing of the rendering intermediate result. Performing anti-aliasing processing on the rendering intermediate result can reduce the sharpness of the edge aliasing of the rendering intermediate result, thereby reducing the difference between the features of the image to be encoded and the rendering intermediate result.

[0189] ii. Feature extraction

[0190] The server can perform feature extraction on the rendering intermediate result (e.g., the Albedo map, which can be output by the image rendering engine or obtained after the anti-aliasing processing described above, which is not limited in particular) by the following steps:

[0191] 1) Obtain a low-frequency map from the rendering intermediate result.

[0192] An exemplary implementation of obtaining a low-frequency map is as follows:

[0193] (1) Divide the rendering intermediate result into multiple image blocks. Exemplarily, the rendering intermediate result can be divided using a preset resolution, for example, 128x128. Exemplarily, the image blocks can also be divided according to an algorithm based on object edges, for example, a geometric partitioning mode (GPM).

[0194] (2) For each image block, calculate its mean value, and then obtain a thumbnail of the image block based on the mean value.

[0195] (3) Enlarge the thumbnails of the multiple image blocks to the size of the rendering intermediate result in an interpolation filtering manner, i.e., obtain a low-frequency map of the rendering intermediate result.

[0196] 2) Extract pixels that meet a preset condition from the rendering intermediate result and the low-frequency map to obtain at least one feature map.

[0197] Two sets of split feature maps (split feature map 1 and split feature map 2) are generated based on the low-frequency map of the rendering intermediate result. The pixel value of the split feature map 1 is the larger one of the pixel values at the corresponding positions in the rendering intermediate result and the low-frequency map of the rendering intermediate result; the pixel value of the split feature map 2 is the smaller one of the pixel values at the corresponding positions in the rendering intermediate result and the low-frequency map of the rendering intermediate result. As shown in Figure 7 ( Figure 7 As shown in the relevant diagrams for feature extraction of the embodiments of the present application, the Albedo map, the thumbnail of the Albedo map, the low-frequency map of the Albedo map, the split feature map 1, and the split feature map 2 are exemplarily shown. The split feature maps (such as the split feature map 1 and the split feature map 2) obtained after feature extraction can provide more fine de-redundancy effects in subsequent steps to improve the encoding performance.

[0198] iii. High-pass filtering

[0199] The high-pass filtering can filter the rendering intermediate result to obtain high-frequency information in the rendering intermediate result. The server can perform high-pass filtering on the rendering intermediate result (e.g., an Albedo map, which can be output by the image rendering engine, can be obtained after the above anti-aliasing processing, or can be obtained after the above feature extraction, which is not limited herein) by the following steps:

[0200] 1) For any one rendering intermediate result, calculate its low-frequency map, and the calculation of the low-frequency map of the rendering intermediate result can use the steps of obtaining the low-frequency map in the above feature extraction steps, which will not be repeated here.

[0201] 2) Subtract the intermediate rendering result and the corresponding low-frequency map to obtain the high-frequency information in the rendering intermediate result.

[0202] The high-pass filtering processing can remove the low-frequency information in the rendering intermediate result, so that only the high-frequency features are adjusted during the redundancy removal processing, so that the redundancy-removed image can still be compatible with the existing video encoding method.

[0203] Optionally, the high-pass filtering processing can also include obtaining the low-frequency flux in the rendering intermediate result, and then removing this part of the low-frequency flux from the rendering intermediate result.

[0204] It should be noted that in one embodiment, the implementation of high-pass filtering is not limited.

[0205] iv. Normal processing: the normal affects the rendering result by interacting with the light direction.

[0206] The server can perform normal processing on the normal map (which can be output by the image rendering engine, or can be obtained after the above anti-aliasing processing, which is not limited herein) by the following steps:

[0207] 1) Divide the rendering intermediate result into multiple image blocks.

[0208] For example, the rendering intermediate result can be divided using a preset resolution, for example, 128x128. For example, the image blocks can also be divided according to an edge-based division algorithm, for example, a geometric partitioning mode (GPM).

[0209] 2) According to the light information in the rendering parameters and the scene depth information that the rendering engine can obtain, set one or more light directions for each image block, and set the light intensity of each light direction.

[0210] 3) For each pixel in the normal map, interpolate the lighting direction and intensity of the pixel according to the lighting directions of the image block and neighboring image blocks that the pixel belongs to.

[0211] 4) For each pixel in the normal map, calculate the pixel value of the corresponding position in the auxiliary image according to the normal direction, lighting direction and intensity of the pixel, using the following formula:

[0212]

[0213] where y is the pixel value of the corresponding position in the auxiliary image, is the normal direction, i is the index of the lighting direction, is the lighting direction of the i-th light, w i is the lighting weight of the i-th light, is the dot product operation of the normal and lighting direction. That is, any pixel value in the auxiliary image is the weighted sum of the non-zero dot product results of the normal and multiple lighting directions.

[0214] Normal processing of the normal map can add lighting features to it, so that the pre-processed normal map increases brightness information and is closer to the image to be encoded.

[0215] 5) Optionally, the embodiment can also use the Albedo map and the normal map together as the rendering intermediate result for encoding and decoding. As shown in Figure 8 ( Figure 8 is the flowchart of the normal processing of the embodiment of the present application) shows that the auxiliary image obtained by the dot product operation (i.e. the output of the dot product operation) can be further multiplied by the Albedo map to obtain the final auxiliary image. The switch is turned on when the rendering intermediate result includes both the Albedo map and the normal map, and turned off when the rendering intermediate result includes only the normal map. In this way, some features in the final auxiliary image are provided by the normal map, and brightness information is added, so that the Albedo can be closer to the image to be encoded.

[0216] In one embodiment, the server can write the above lighting intensity and lighting direction into the code stream, so that the terminal device can obtain these two information by decoding the code stream and use them in the subsequent steps, thereby improving the decoding performance.

[0217] It should be noted that the embodiment of the present application can also preprocess other rendering intermediate results to reduce the difference between the auxiliary image and the image to be encoded, thereby providing more fine de-redundancy effect to improve the encoding performance.

[0218] Optionally, the embodiment of the present application further provides a pre-processing identifier, which is used to indicate the pre-processing manner adopted in step 402, such as at least one of the anti-aliasing processing, feature extraction, or high-pass filtering. The pre-processing identifier can also be written into the code stream, so as to be obtained by the terminal device through decoding the code stream and used in the processing of the terminal device.

[0219] Step 403: performing de-redundancy processing on the to-be-encoded image according to the auxiliary image to obtain a de-redundancy image.

[0220] In an embodiment, the server can perform de-redundancy processing on the to-be-encoded image according to the auxiliary image to obtain a de-redundancy image.

[0221] The de-redundancy processing adopted by the server can include:

[0222] (1) When there is only one auxiliary image, calculating the difference between the auxiliary image and the to-be-encoded image (e.g., the difference between the pixel values at the corresponding positions), i.e., calculating the pixel value difference at the corresponding positions in the auxiliary image and the to-be-encoded image, so as to form the de-redundancy image.

[0223] (2) When there are multiple auxiliary images, calculating the difference between one of the auxiliary images and the to-be-encoded image, i.e., calculating the pixel value difference at the corresponding positions in the auxiliary image and the to-be-encoded image, and then calculating the difference between the pixel value in another auxiliary image and the pixel value difference at the corresponding positions, and so on, until all the auxiliary images are calculated, and the final pixel difference forms the de-redundancy image.

[0224] In another embodiment, the server can first obtain an adjusted auxiliary image, which is closer to the to-be-encoded image (i.e., the similarity between the adjusted auxiliary image and the to-be-encoded image is higher) than the auxiliary image; and then perform de-redundancy processing on the to-be-encoded image according to the adjusted auxiliary image to obtain a de-redundancy image. In this way, since the similarity between the adjusted auxiliary image and the to-be-encoded image is higher, the de-redundancy processing on the to-be-encoded image according to the adjusted auxiliary image can further reduce the redundant information in the to-be-encoded image, and the de-redundancy effect is better, thereby further improving the encoding performance.

[0225] In an embodiment, the adjustment method adopted by the server on the auxiliary image can include: obtaining at least one set of adjustment parameters according to the auxiliary image and the to-be-encoded image; constructing at least one adjustment function according to the at least one set of adjustment parameters; and adjusting the pixels of the corresponding auxiliary image according to the at least one adjustment function to obtain an adjusted auxiliary image.

[0226] In another embodiment, the adjustment method employed by the server on the auxiliary image can include: dividing the auxiliary image into a plurality of auxiliary image blocks; for any one auxiliary image block, obtaining at least one set of adjustment parameters according to the auxiliary image block and the corresponding block in the image to be encoded; constructing at least one adjustment function according to the at least one set of adjustment parameters; and adjusting the pixels of the corresponding auxiliary image block respectively according to the at least one adjustment function to obtain an adjusted auxiliary image block, and the plurality of adjusted auxiliary image blocks constitute an adjusted auxiliary image.

[0227] The server can perform the above adjustment on the entire frame of the auxiliary image, or can first divide the auxiliary image into a plurality of auxiliary image blocks as described above, and then perform the above adjustment on each auxiliary image block to obtain an adjusted auxiliary image. The division mode employed by the server for dividing the auxiliary image can include regular division or irregular division. The regular division can divide the auxiliary image into 9 blocks, 16 blocks, etc., and the irregular division can be division according to the geometric boundaries of the content in the auxiliary image. The above division modes can be encoded into the code stream. In one embodiment, the server can divide according to the edges of the object, for example, as shown in Figure 5 ( Figure 5 a geometric partitioning mode (GPM) algorithm to divide the image into "triangle + trapezoid" regions, so that the terminal device only needs to know the angle and offset of the division line to successfully decode. It can be seen that the GPM algorithm can divide the image and provide more detailed adjustment processing. Based on this, the server can write the division information (including the correspondence between the auxiliary image blocks and the adjustment parameters) into the code stream, so that the terminal device can locate and obtain the auxiliary image block based on the division information, and then adjust the auxiliary image block based on the corresponding adjustment parameters, thereby bringing better and more detailed detail adjustment effect and improving the encoding performance.

[0228] In one embodiment, the adjustment function constructed by the server can include any one of the following functions or a combination of multiple functions: a proportional function, a linear function, a polynomial function, a power function, or a function represented by a neural network. For example, a power function (e.g., y = x n ), or a proportional function (e.g., y = kx), or a combination of a power function and a proportional function (e.g., y = kx n ), or a function represented by a neural network is a function that takes a single pixel value as input and outputs a single pixel value, with a neural network as an intermediate layer. It should be noted that the foregoing exemplary describes several forms of functions, but does not limit the adjustment function. The type and composition of the adjustment function are not limited in the embodiments of the present application.

[0229] According to the function to be constructed, the adjustment parameter corresponding to the function can be determined, for example, y = x n The corresponding transformation parameter is n, and the corresponding transformation parameter of y = kx is k, y = kx n The corresponding transformation parameters are k and n. In an embodiment, the adjustment of the server to the auxiliary image can be pixel-level, that is, all pixel values in the auxiliary image can be adjusted by the same adjustment function, but the adjustment parameters corresponding to each pixel value can be different, or each pixel value in the auxiliary image can be adjusted by different adjustment functions, and the adjustment parameters corresponding to each pixel value can be different. For example, the auxiliary image includes 64 pixels, corresponding to 64 pixel values, which can correspond to 64 groups of adjustment parameters, that is, the adjustment parameters of the 64 pixel values are different; or the 64 pixel values can correspond to 2 groups of adjustment parameters, that is, the adjustment parameters of part of the 64 pixel values are the same, and the adjustment parameters of the other part of the pixel values are different; or the 64 pixel values can correspond to 3 groups or more groups (less than 64 groups) of adjustment parameters, that is, the 64 pixel values are divided into different sets, and the adjustment parameters of the pixel values in the same set are the same, and the adjustment parameters of the pixel values in different sets are different. In other words, any group of adjustment parameters can correspond to one or more pixel values in the auxiliary image.

[0230] When a group of adjustment parameters is determined, an adjustment function can be constructed based on the group of adjustment parameters, that is, the adjustment function is used to adjust the pixels of the corresponding auxiliary image or auxiliary image block. When the corresponding pixel value is input into the adjustment function, the output of the adjustment function is regarded as the adjusted pixel value of the corresponding pixel. When all pixel values in the auxiliary image are adjusted according to the foregoing process, all the adjusted pixel values form an adjusted auxiliary image.

[0231] In an embodiment, the server can determine the adjustment parameter of the first pixel (the first pixel can be any pixel in the auxiliary image) in the auxiliary image by comparing the code stream size. For example, a plurality of groups of adjustment parameters are preset, and then a plurality of adjustment functions are constructed, and the first pixel is adjusted by using the plurality of adjustment functions respectively, and a plurality of adjusted pixel values of the first pixel can be obtained. The residuals of the plurality of adjusted pixel values of the first pixel and the pixel values of the corresponding pixels in the image to be encoded are calculated respectively, and a plurality of residuals are obtained. The plurality of residuals are encoded respectively, and the code stream size of the plurality of residual encodings is measured. Exemplarily, the adjustment parameter corresponding to the smallest code stream can be determined as the adjustment parameter of the first pixel, and a preset code stream size threshold can be set, and the adjustment parameter meeting the preset code stream size threshold is selected as the adjustment parameter of the first pixel.

[0232] In one embodiment, in the rendering result, the image boundary can be derived from the geometric boundary of the object or the boundary generated by the lighting, and the matching degree of the auxiliary image and the rendering result can be different on both sides of the boundary, for example, the relationship between the Albedo map and the color of the rendering result is usually the combination of the Albedo map, the lighting, and the material attribute (for example, roughness / slippery). Different objects on both sides of the boundary can be far apart in space (for example, object A blocks object B, but the spatial distance between objects A and B is far apart), and the spatial distance causes the difference in lighting. Under different lighting, the mapping relationship between the Albedo map and the rendering result is different. For example, the Albedo map can be multiplied by 0.1 when it is dark, and multiplied by 2 when it is bright. In addition, the difference in material between the Albedo map and the rendering result is also similar. Therefore, the adjustment function used for the pixel value near the boundary should also be different.

[0233] By selecting the appropriate adjustment function and adjustment parameter for each pixel value in the auxiliary image, the matching degree of the region-based feature of the auxiliary image in the region can be finely adjusted, so that the adjusted auxiliary image after the adjustment function is adjusted has a higher similarity with the image to be encoded, and thus in the subsequent redundancy removal processing, more redundant information between the auxiliary image and the image to be encoded can be reduced, and the encoding performance is improved.

[0234] As described above, the auxiliary image of the image to be encoded can include one or more, and therefore the server can use the above method to adjust each auxiliary image respectively, so as to obtain one or more adjusted auxiliary images of the image to be encoded.

[0235] In another embodiment, the server can first perform frequency domain transformation on the auxiliary image to obtain an auxiliary image frequency map; then perform redundancy removal processing on the image frequency map to be encoded according to the auxiliary image frequency map to obtain a redundancy removed image frequency map; and then perform inverse frequency domain transformation on the redundancy removed image frequency map to obtain a redundancy removed image. That is, the server performs adjustment in the frequency domain, instead of applying different adjustment functions to different regions of the same image, but applying different adjustment functions to different frequency details of the image (equivalent to adjusting according to the frequency in the original image domain). Although there are differences in principle, the same redundancy removal effect can be achieved by using the information of the rendering intermediate image in the frequency domain. Moreover, by adjusting the details according to the frequency, since the signal of any frequency will affect the pixel value of the entire image, the generated image can have better spatial continuity, which can further improve the encoding and decoding performance in some scenarios.

[0236] Step 404, encoding the redundancy removed image to obtain a code stream.

[0237] The server can encode the de-redundant image into a bitstream by a standard video encoder, which can be implemented according to an industry video compression standard, for example, the standard can be ITU-T H.264, H.265 / HEVC, H.266 / VVC, AVS2, AVS3 standard, or an extension of such a standard. The technology of the present application is not limited to any specific coding standard.

[0238] In an embodiment, the bitstream can include an adjustment parameter for adjusting the auxiliary image.

[0239] In an embodiment, the bitstream can further include a correspondence between the at least one set of adjustment parameters and the auxiliary image, the auxiliary image block, or the pixel value in the auxiliary image or the auxiliary image block.

[0240] For example, the server can write the adjustment parameter value in the bitstream, and then the terminal device can input the aforementioned adjustment parameter value according to a pre-agreed adjustment function, thereby constructing the final adjustment function.

[0241] In an embodiment, the bitstream can include a constructed adjustment function.

[0242] In an embodiment, the bitstream can include an index related to the adjustment parameter or the adjustment function. The terminal device can obtain the adjustment parameter or the adjustment function corresponding to the index according to a pre-agreed correspondence, and then obtain the final adjustment function. It should be noted that the embodiments of the present application can transmit the adjustment parameter or the adjustment function in multiple ways, which are not limited herein.

[0243] In an embodiment, the bitstream further includes the rendering intermediate result corresponding to the image to be encoded, or the auxiliary image, or the adjusted auxiliary image. The server can directly transmit the rendering intermediate result corresponding to the image to be encoded, or the auxiliary image, or the adjusted auxiliary image to the terminal device, so that the terminal device directly decodes the stream to obtain the aforementioned result without rendering processing, which can improve the performance of the terminal device and reduce the rendering capability requirement of the terminal device.

[0244] Figure 10 For the flowchart of the encoding end provided in the embodiments of the present application, it should be understood that the flowchart is only an example of combination of the aforementioned embodiments of the encoding process, and the aforementioned embodiments of the encoding process can also be combined in multiple ways, but the embodiments after multiple combinations should be within the protection scope of the present application. The encoding end (corresponding to the server) encodes a plurality of frames, and each frame of compressed image data and adjustment parameter is written into a bitstream. The encoding process for one frame of image includes the following steps:

[0245] 1. Obtain the rendering intermediate result (corresponding to step 401)

[0246] The rendering intermediate result can be, for example, a normal map, an albedo map, etc.

[0247] 2. Preprocessing the rendering intermediate result to obtain an auxiliary image (corresponding to step 402)

[0248] The preprocessing includes, for example, anti-aliasing processing, feature extraction, high-pass filtering, and can also include normal processing for a normal map.

[0249] 3. Calculating an adjustment parameter (corresponding to the step of obtaining the adjustment parameter in step 403)

[0250] The encoding end can determine the adjustment parameter of a first pixel (which can be any pixel in the auxiliary image) in the auxiliary image by comparing the sizes of code streams. For example, a plurality of sets of adjustment parameters are preset, and then a plurality of adjustment functions are constructed, and the first pixel is adjusted by using the plurality of adjustment functions respectively, so that a plurality of adjusted pixel values of the first pixel can be obtained. Residuals can be obtained by respectively calculating the residuals between the plurality of adjusted pixel values of the first pixel and the pixel values of the corresponding pixels in the image to be encoded. The plurality of residuals are encoded respectively, and the sizes of the code streams after the plurality of residuals are encoded are measured, and the adjustment parameter corresponding to the smallest code stream is determined as the adjustment parameter of the first pixel. The same method can be used to obtain the adjustment parameters of all pixels in the auxiliary image, and the adjustment parameters are written into the code stream and transmitted to the decoding end.

[0251] 4. Adjustment (corresponding to the step of obtaining the adjusted auxiliary image based on the obtained adjustment parameter in step 403)

[0252] The encoding end adjusts the pixel values of the auxiliary image according to the adjustment parameter to better match the image to be encoded. That is, a function (also called an adjustment function) is constructed for each pixel value of the auxiliary image according to the adjustment parameter, and then each pixel in the auxiliary image is applied with the corresponding adjustment function, so that the pixel value of the pixel is used as the input of the adjustment function, and the output value of the adjustment function is used as the adjusted value of the pixel, thereby forming an adjusted auxiliary image.

[0253] Then, the pixel values at corresponding positions of the adjusted auxiliary image and the image to be encoded are subtracted to obtain a de-redundancy image, which is input into an encoding module.

[0254] 5. Encoding (corresponding to step 404)

[0255] The encoding end receives the de-redundancy image, encodes the de-redundancy image using a standard video encoder to obtain a code stream corresponding to the de-redundancy image, and transmits the code stream to the decoding end.

[0256] In the embodiments of the present application, the encoding end generates a code stream and transmits it to the decoding end. An exemplary structure of a code stream can be as follows:

[0257] 1) Data header (optional): used to record the metadata of the code stream.

[0258] 2) Block information (optional): used to make the decoding end perform the same image blocking operation as the encoding end. In an embodiment, there is an operation of dividing the image into blocks by area and then further processing; in an embodiment, the blocking mode of the image can be determined by the encoding end, and at this time, the blocking information can be transmitted to the decoding end.

[0259] 3) Adjustment parameters (optional): used to control the adjustment process of the decoding end. The decoding end adjusts the pixel value of the auxiliary image based on the adjustment parameters to restore the image information subtracted by the encoding end.

[0260] 4) De-redundancy image (mandatory): the de-redundancy image generated after the encoding end removes the redundancy, and the code stream data generated after the video encoder encodes. The decoding end can be used to generate the final display reconstructed image after decoding the code stream.

[0261] 5) Light direction and light intensity (optional): used for normal processing of the normal map when the rendering intermediate result is a normal map. The decoding end performs normal processing on the normal map based on the light direction and light intensity.

[0262] In the embodiments of the present application, the auxiliary image is obtained by preprocessing the rendering intermediate result, so that the auxiliary image has higher similarity with the image to be encoded, and thus more redundant information between the auxiliary image and the image to be encoded can be reduced in the de-redundancy processing, and the encoding performance is improved.

[0263] Figure 9 A flowchart of a process 900 of an image decoding method provided by the embodiments of the present application is shown. The process 900 can be performed by the terminal device described above. The process 900 is described as a series of steps or operations, and it should be understood that the process 900 can be performed in various orders and / or simultaneously, and is not limited to the execution order shown. The process 900 can include: Figure 9

[0264] Step 901, receiving a code stream.

[0265] Step 902, obtaining a de-redundancy image according to the code stream.

[0266] The terminal device can decode the code stream by a standard video decoder to obtain the de-redundancy image. The standard video decoder can be implemented according to an industry video compression standard, for example, the standard can be ITU-T H.264, H.265 / HEVC, H.266 / VVC, AVS2, AVS3 standard, or an extension of such a standard. The technology of the present application is not limited to any specific decoding standard. ​

[0267] Step 903, obtaining a rendering intermediate result.

[0268] In an embodiment, the terminal device renders the three-dimensional scene to obtain a rendering intermediate result. For example, refer to step S303 described above. Figure 3a

[0269] In an embodiment, the terminal device can also obtain the rendering intermediate result from the bitstream.

[0270] The de-redundant image and the rendering intermediate result can refer to an entire image, or a partial image of a frame of image, for example, an image block, an ROI, etc., and the size and shape of the partial image are not limited.

[0271] Step 904, pre-processing the rendering intermediate result to obtain an auxiliary image.

[0272] In an embodiment, corresponding to the operation of the server at the encoding end, the terminal device at the decoding end can also pre-process the rendering intermediate result to obtain one or more auxiliary images, and then perform subsequent processing. The foregoing pre-processing can include at least one of anti-aliasing processing, feature extraction, or high-pass filtering, and can also include normal processing of a normal map. For pre-processing, refer to step 402 of the embodiment described above, which will not be described here. Figure 4

[0273] In addition, in an embodiment, the terminal device can also obtain the foregoing pre-processing identifier from the bitstream, and then pre-process the rendering intermediate result according to the pre-processing mode indicated by the pre-processing identifier to obtain an auxiliary image.

[0274] Step 905, obtaining a reconstructed image according to the auxiliary image and the de-redundant image.

[0275] In an embodiment, obtaining a reconstructed image according to the auxiliary image and the de-redundant image can include: when the auxiliary image is only one, reconstructing (for example, summing the pixel values of corresponding positions) based on the auxiliary image and the de-redundant image, that is, calculating the sum of the pixel values at corresponding positions in the auxiliary image and the de-redundant image, thereby composing a reconstructed image.

[0276] In an embodiment, obtaining a reconstructed image according to the auxiliary image and the de-redundant image can include: when the auxiliary image is multiple, summing one of the auxiliary images and the de-redundant image, that is, calculating the sum of the pixel values at corresponding positions in the auxiliary image and the de-redundant image, and then calculating the sum of the pixel values in another auxiliary image and the sum of the pixel values at the foregoing corresponding positions, and so on, until all auxiliary images are calculated, and the final pixel sum composes a reconstructed image.

[0277] ​​In one embodiment, the terminal device can first obtain the adjusted auxiliary image, and then reconstruct the reconstructed image according to the adjusted auxiliary image and the de-redundancy image.

[0278] In one embodiment, the adjustment method adopted by the terminal device on the auxiliary image can include: obtaining at least one set of adjustment parameters according to the code stream; constructing at least one adjustment function according to the at least one set of adjustment parameters; and adjusting the pixels of the auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image. Optionally, one or more pixels in the auxiliary image correspond to one set of adjustment parameters in the at least one set of adjustment parameters, and the pixels corresponding thereto in the auxiliary image are adjusted according to the at least one adjustment function to obtain the adjusted auxiliary image.

[0279] In another embodiment, the adjustment method adopted by the terminal device on the auxiliary image can include: obtaining a set of adjustment parameters corresponding to an auxiliary image block according to the code stream, the set of adjustment parameters corresponding to the auxiliary image block or one or more pixels in the auxiliary image block, the auxiliary image including a plurality of auxiliary image blocks; constructing at least one adjustment function according to the at least one set of adjustment parameters; adjusting the pixels of the corresponding auxiliary image block according to the at least one adjustment function to obtain the adjusted auxiliary image block corresponding to the auxiliary image block; and obtaining the adjusted auxiliary image according to the plurality of adjusted auxiliary image blocks. Optionally, the set of adjustment parameters corresponding to the auxiliary image block can be obtained according to the corresponding relationship between the auxiliary image block and the adjustment parameter in the code stream.

[0280] Optionally, the terminal device can further perform block processing on the auxiliary image. In one embodiment, the block processing manner is determined according to a block mode, which can be preset or obtained from the code stream. The specific block processing manner and the block mode can refer to the description of the server above.

[0281] The adjustment parameters and the adjustment function can refer to the description of the step 403 in the embodiment shown in Figure 4 The related description in the step 403 of the embodiment shown in

[0282] As described above, the auxiliary image can include one or more, and therefore the terminal device can adjust each auxiliary image respectively by using the above method to obtain one or more adjusted auxiliary images.

[0283] In one possible implementation, the code stream further includes the rendering intermediate result or the auxiliary image. The terminal device directly decodes the stream to obtain the rendering intermediate result or the auxiliary image, without the need for rendering processing, which can improve the decoding performance. Alternatively, the terminal device directly decodes the stream to obtain the adjusted auxiliary image, without the need for rendering and adjustment processing on the auxiliary image, which can improve the decoding efficiency of the terminal device.

[0284] Figure 11For the flowchart of the decoding end of an embodiment of the present application, it should be understood that Figure 11 The above is only an exemplary description of one combination of the embodiments of the decoding method, and the embodiments of the decoding method can also have various combinations, but the embodiments after the various combinations should be within the protection scope of the present application. The decoding end (corresponding to a terminal device) decodes a code stream, reconstructs a plurality of frames, and the code stream includes compressed image data and adjustment parameters of each frame. The decoding process for a frame of image includes the following steps:

[0285] 1. Obtain a rendering intermediate result (corresponding to step 903)

[0286] The rendering intermediate result can be, for example, a normal map, an albedo map, etc.

[0287] 2. Preprocess the rendering intermediate result to obtain an auxiliary image (corresponding to step 904)

[0288] 3. Decode (corresponding to step 902)

[0289] The decoding end receives a code stream, and uses a standard video decoder to obtain a de-redundancy image and adjustment parameters.

[0290] 4. Adjust (corresponding to the embodiment of adjusting the auxiliary image in step 905)

[0291] The decoding end constructs an adjustment function according to the adjustment parameters, and adjusts the pixel values in the auxiliary image through the adjustment function to obtain an adjusted auxiliary image, so as to match the adjustment made by the encoding end on the auxiliary image.

[0292] 5. Reconstruct (corresponding to the embodiment of reconstructing in step 905)

[0293] The pixel values at the corresponding positions of the adjusted auxiliary image and the de-redundancy image are added to restore the redundant information removed by the encoding end, and a reconstructed image is obtained.

[0294] In the embodiments of the present application, the rendering intermediate result is preprocessed in the server to obtain an auxiliary image. Because the auxiliary image has a high similarity with the image to be encoded, the de-redundancy image obtained by performing de-redundancy processing on the image to be encoded according to the auxiliary image can reduce more redundant information in the image to be encoded. When such a de-redundancy image is encoded into a code stream and transmitted to a terminal device, the effect of saving the code stream can be achieved. The terminal device can obtain a reconstructed de-redundancy image by decoding the stream, and obtain an auxiliary image by preprocessing the rendering intermediate result. The reconstructed image can be reconstructed by combining the auxiliary image, and the decoding performance can also be improved.

[0295] In the following, several specific embodiments are used to describe the encoding method and the decoding method. Figure 4 and Figure 9The technical solutions of the method embodiments are described in detail.

[0296] Figure 12 A schematic diagram of an end-cloud codec architecture of the present embodiment is shown in the figure. In the graphics rendering engine, the inherent color of each object is often described in terms of albedo, and in the process of calculating the rendered image, an albedo map of the rendered object is generated and further calculations and rendering are carried out based on the albedo map. As a description of the inherent color of the object, the albedo map often has great visual similarity to the final rendering result and contains a large amount of information about the final rendering result. In the cloud gaming scenario, the albedo map synchronized with the cloud side can be rendered by the graphics rendering engine at the end side at a lower cost through methods such as pre-deploying resources. Therefore, the present embodiment uses the end-cloud synchronized albedo map to assist image coding, thereby achieving the effect of improving the coding performance of the rendered image.

[0297] As shown in Figure 12 , in the end-cloud codec architecture, the de-redundancy processing module of the cloud side (encoding end) includes Figure 10 the three modules of obtaining an auxiliary image, calculating an adjustment parameter, and adjusting in the embodiment shown; and the reconstruction module of the end side (decoding end) includes Figure 11 the two modules of obtaining an auxiliary image and adjusting in the embodiment shown. The rendering parameters include various parameters used for rendering, the positions and color vectors of all light sources, the position vector of the player or observer, the sampling method of each texture, the position coordinates of each scene object, the motion trajectory of the moving object, the skeletal animation parameters, and the like. Some of the rendering parameters are all at the end side, some are all at the cloud side, and some are common to both the end side and the cloud side. Therefore, the synchronization of the rendering parameters is required before each frame is rendered. The present embodiment does not limit the synchronization method and synchronization content of the rendering parameters.

[0298] The encoding process of the cloud side includes the following steps:

[0299] 1. Rendering parameter synchronization and acquisition

[0300] a) Transmission of rendering parameters

[0301] i. Write the rendering parameters required to be provided to the end side in the end-cloud synchronization into the code stream and transmit them to the end side.

[0302] b) Acquisition of rendering parameters

[0303] i. Decode the rendering parameter code stream uploaded by the end side as the rendering parameters.

[0304] ii. Load the rendering assets pre-existing in the hard disk / in-memory as the rendering parameters.

[0305] 2. Generation of rendering results

[0306] a) input the rendering parameters obtained in step 1 into a graphics rendering engine (for example, V-Ray, Unreal, Unity, and other applications with rendering capabilities), and the graphics rendering engine performs a rendering process according to the rendering parameters, renders objects in the scene, and performs rendering shading, lighting calculation, etc., thereby generating a rendering result as an image to be encoded.

[0307] 3. Generating a rendering intermediate result

[0308] The rendering intermediate result in this example refers to an Albedo map. Generating the Albedo map includes the following steps:

[0309] i. input the rendering parameters obtained in step 1 into a graphics rendering engine (for example, V-Ray, Unreal, Unity, and other applications with rendering capabilities), and the graphics rendering engine performs a rendering process according to the rendering parameters, renders objects in the scene, and when determining the color of a pixel, takes the Albedo map color value of the point on the object corresponding to the pixel as the color of the pixel, thereby obtaining the Albedo map in the rendering intermediate result.

[0310] 4. De-redundancy processing

[0311] a) obtaining an auxiliary image

[0312] Further preprocessing is performed on the Albedo map generated in step 3:

[0313] i. Anti-aliasing processing

[0314] The cloud side can use at least one of the following methods:

[0315] 1) Anti-aliasing processing is performed on the Albedo map in the same way as when the graphics rendering engine generates the image to be encoded.

[0316] 2) Blurring or other operations are performed on the Albedo map, thereby reducing the sharpness of the edge aliasing.

[0317] ii. Feature extraction

[0318] The cloud side can perform feature extraction on the Albedo map (which can be output by the image rendering engine or obtained after the above anti-aliasing processing, without specific limitation) by the following steps:

[0319] 1) The Albedo map is divided into multiple Albedo image blocks at a certain resolution (for example, 128x128).

[0320] 2) For each Albedo image block, the mean value is calculated, and a thumbnail of the Albedo image block is obtained based on the mean value.

[0321] 3) The thumbnails of the plurality of Albedo image blocks are enlarged to the size of the Albedo map in an interpolation filtering manner, i.e. the low-frequency map of the Albedo map is obtained.

[0322] 4) Two groups of split feature maps are generated according to the low-frequency map of the Albedo map. The pixel value of the split feature map 1 is the larger one of the pixel values at the corresponding positions in the Albedo map and the low-frequency map of the Albedo map. The pixel value of the split feature map 2 is the smaller one of the pixel values at the corresponding positions in the Albedo map and the low-frequency map of the Albedo map. As shown in the following figure, exemplary thumbnails, the low-frequency map of the Albedo map, the split feature map 1 and the split feature map 2 are shown. Figure 7 After feature extraction, the split feature maps (such as the split feature map 1 and the split feature map 2 mentioned above) can provide more refined de-redundancy effects in subsequent steps to improve the encoding performance.

[0323] iii. High-pass filtering

[0324] The cloud side can perform high-pass filtering on the Albedo map (which can be output by the image rendering engine, or obtained after the anti-aliasing processing, or obtained after the feature extraction, which is not specifically limited here) by the following steps:

[0325] 1) Calculate the low-frequency map for each Albedo map.

[0326] The calculation of the low-frequency map of the Albedo map can adopt the steps of obtaining the low-frequency map in the feature extraction step, which is not repeated here.

[0327] 2) Subtract the corresponding low-frequency map from each auxiliary image to obtain the final auxiliary image.

[0328] The high-pass filtering processing can remove the low-frequency information in the Albedo image, so that only the high-frequency features are adjusted during the de-redundancy processing, so that the de-redundancy image can still be compatible with existing video encoding methods.

[0329] b) Calculate the adjustment parameter

[0330] i. Divide the auxiliary image and the image to be encoded into a plurality of image blocks at a certain resolution (for example, 128x128).

[0331] ii. Obtain the appropriate adjustment function for each auxiliary image block. In this embodiment, the appropriate adjustment function can be a proportional function, such as:

[0332] f(x) = kx

[0333] That is, scale each pixel value in the auxiliary image block by k times.

[0334] The suitable adjustment function form can make the auxiliary image mapped by the adjusted function have a higher similarity with the image to be encoded. The similarity can be obtained by calculating the residual of the two images, encoding the residual as an image, and measuring the size of the encoded code stream. For example, a plurality of k values can be enumerated, and then the optimal k value is selected as the optimal adjustment parameter of the auxiliary image.

[0335] iii. The k values corresponding to different pixel values in the auxiliary image constitute a plurality of sets of adjustment parameters in this embodiment. The plurality of sets of adjustment parameters are transmitted to the adjustment module and also written into the code stream and transmitted to the terminal side.

[0336] c) Adjustment

[0337] i. For each pixel value in the auxiliary image map, the adjustment parameter of the pixel value is obtained by interpolation filtering according to the adjustment parameters of the image block and the adjacent image block to which the pixel value belongs stored in the adjustment parameter.

[0338] In order to save the code stream, the adjustment parameters of each pixel value in the auxiliary image cannot be all written into the code stream and brought to the terminal side. As an example, the adjustment parameters of the pixel value located at the center of each auxiliary image or each auxiliary image block in the auxiliary image can be written into the code stream, and the adjustment parameters of other pixel values of the auxiliary image or the auxiliary image block in the auxiliary image are not written into the code stream. The terminal side can decode the stream to obtain the adjustment parameters of the pixel value located at the center of each auxiliary image or each auxiliary image block in the auxiliary image, and then use the interpolation method to obtain the adjustment parameters of other pixel values of the auxiliary image or the auxiliary image block in the auxiliary image. The reason why interpolation is used instead of directly using the adjustment parameters of the pixel value located at the center of the image block is mainly to prevent the edge of the image block from appearing abrupt.

[0339] ii. The adjustment function of the pixel value is constructed according to the function coefficient of the pixel value.

[0340] iii. The changed pixel value is obtained by mapping the pixel value through the corresponding adjustment function, and the adjusted auxiliary image is finally obtained.

[0341] d) Generating a de-redundancy image

[0342] The pixel values at the corresponding positions in the adjusted auxiliary image output by the adjustment module and the image to be encoded are subtracted. When there are a plurality of auxiliary images, the residual of one of the adjusted auxiliary images and the image to be encoded is calculated, that is, the difference between the pixel values at the corresponding positions in the adjusted auxiliary image and the image to be encoded is calculated, and then the difference between the pixel value in another adjusted auxiliary image and the pixel value at the corresponding position is calculated. In this way, all auxiliary images are calculated, and the final pixel difference constitutes a de-redundancy image.

[0343] 5. Encoding

[0344] a) The de-redundancy image generated by the de-redundancy processing module is encoded using the standard video encoding process to obtain the de-redundancy image code stream.

[0345] 6. The code stream is packaged and transmitted to the terminal side. The code stream includes:

[0346] a) The adjustment parameters used in the adjustment.

[0347] b) The code stream of the de-redundancy image generated by the video encoding.

[0348] The decoding process at the terminal side includes the following steps:

[0349] 1. Rendering parameter synchronization and acquisition

[0350] a) Transmission of rendering parameters

[0351] i. The rendering parameters needed to be provided to the cloud side in the terminal-cloud synchronization are written into the code stream and transmitted to the cloud side.

[0352] b) Acquisition of rendering parameters

[0353] i. The rendering parameter code stream distributed by the cloud side is decoded as the rendering parameters.

[0354] ii. The pre-existing rendering assets in the hard disk / memory are loaded as the rendering parameters.

[0355] 2. Generation of rendering intermediate results

[0356] This step is consistent with step 3 of the cloud side.

[0357] 3. Code stream decoding, which can obtain:

[0358] a) The adjustment parameters used in the adjustment.

[0359] b) The de-redundancy image

[0360] 4. Reconstruction

[0361] a) Acquisition of auxiliary images

[0362] This step is consistent with step 4.a) of the cloud side.

[0363] b) Adjustment

[0364] According to the adjustment parameters obtained from the decoded code stream, the same adjustment as step 4.c) of the cloud side is performed to obtain the adjusted auxiliary image.

[0365] c) Generation of reconstructed images

[0366] The pixel values of the corresponding positions in the decoded de-redundancy image and the adjusted auxiliary image are added, and when there are multiple auxiliary images, one of the adjusted auxiliary images and the de-redundancy image are summed, that is, the sum of the pixel values at the corresponding positions in the adjusted auxiliary image and the de-redundancy image is calculated, and then the sum of the pixel values in another adjusted auxiliary image and the pixel values at the corresponding positions is calculated, and so on, until all the auxiliary images are calculated, and the final pixel sum constitutes the reconstructed image.

[0367] The Albedo map generated by the end-cloud collaborative rendering is used as an auxiliary image in this embodiment, and only the adjustment parameters need to be transmitted, which can reduce the amount of information of the video directly transmitted in the cloud game scenario, and therefore the code stream consumption is reduced compared with directly encoding the video. The pixel values of the auxiliary image are adjusted by using the adjustment parameters in the adjustment function, which are calculated from the auxiliary image and the image to be encoded. In the adjustment parameters, the redundant information of the image to be encoded is actually covered, and these information is transmitted to the decoding end in the form of adjustment parameter code stream, so that the coding performance of the Albedo map used for auxiliary video coding is improved. Exemplarily, Figure 13 The exemplary comparison diagram of the rendering results of this embodiment, wherein the rendering results include the rendering results of the game engine, the Albedo map corresponding to the rendering results, and the rendering results subjected to the de-redundancy processing of the application.

[0368] In another embodiment, the normal map can be used instead of the Albedo map as an auxiliary image in the embodiment shown. Figure 12 In this embodiment, the end side and the cloud side both have the normal map as a rendering asset; the rendering parameters include the three-dimensional model of the scene, the lighting information, the normal map, and the camera parameter, and the only rendering parameter that needs to be synchronized is still the camera parameter of the user.

[0369] The difference between this embodiment and the embodiment shown in the above Figure 12 is that the normal map in the generated rendering intermediate result is used as an auxiliary image in this embodiment.

[0370] Optionally, in an embodiment, the normal map can also be preprocessed in the step of obtaining the auxiliary image on the cloud side to obtain the final auxiliary image. The preprocessing of the normal map includes at least one of the anti-aliasing processing, feature extraction, high-pass filtering, and normal processing in the embodiment shown in the above Figure 12 The normal map is affected by the lighting direction to affect the rendering result. The normal processing can refer to the part of the normal processing in step 402 in the above Figure 4 .

[0371] In addition to the above differences, the other steps of this embodiment can refer to the related steps in the embodiment shown in the above Figure 12 , which will not be described here.

[0372] Figure 14 An end-to-cloud encoding and decoding architecture of an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the end-to-cloud encoding and decoding architecture of the embodiment of the present application comprises a server and a terminal. The server comprises a rendering engine, a frequency domain adjuster, a frequency domain de-redunder, an image encoder, and a cloud-to-end decoder. The terminal comprises an image decoder and a rendering engine. Figure 12 The difference between the embodiment shown in FIG. 1 and the embodiment shown in FIG. 2 is that, in the embodiment shown in FIG. 2, the rendering engine of the server does not generate a rendering result, but generates a rendering intermediate result. The rendering intermediate result is encoded by the image encoder of the server, and then transmitted to the terminal. Thus, the code stream further comprises the code stream of the rendering intermediate result. The terminal can obtain the rendering intermediate result by decoding the code stream.

[0373] In addition to the above difference, the other steps of the embodiment of the present application can refer to the related steps in the embodiment shown in FIG. 1, which will not be described here again. Figure 12

[0374] The embodiment of the present application splits the rendering image into a rendering intermediate result and a de-redundant image after removing the details of the rendering intermediate result, and encodes and transmits the two to the terminal. In the case where both the rendering intermediate result and the rendering result need to be transmitted to the terminal, the embodiment of the present application can reduce the bandwidth.

[0375] Figure 15 An end-to-cloud encoding and decoding architecture of an embodiment of the present application is shown in FIG. 1. In the field of image processing, any image has its corresponding frequency domain representation, and the frequency domain representation of an image can still be represented by an image (hereinafter referred to as a frequency image) of the same resolution as the image. The frequency image of the auxiliary image and the frequency image of the image to be encoded still have similar image features, and the image features of the image to be encoded can be removed in the frequency domain by the embodiment of the present application.

[0376] Based on this, the server can perform frequency domain transformation on the auxiliary image and the image to be encoded to obtain the frequency image of the auxiliary image and the frequency image of the image to be encoded. Then, the frequency image of the image to be encoded is de-redundant processed according to the frequency image of the auxiliary image to obtain a de-redundant image frequency image. Finally, the de-redundant image frequency image is inversely transformed in the frequency domain to obtain a de-redundant image.

[0377] That is, the server performs adjustment in the frequency domain, instead of applying different adjustment functions to different regions of the same image, but applying different adjustment functions to different frequency details of the image (performing blocking in the frequency domain and adjusting by block, which is equivalent to adjusting by frequency level in the original image domain (i.e. the image domain before frequency domain processing)). Although there is a difference in principle, the adjustment of the auxiliary image in the frequency domain can also utilize the information of the rendering intermediate image to achieve the de-redundant effect. Moreover, the adjustment of the frequency details can produce an image with better spatial continuity, as any frequency signal will affect the pixel value of the entire image, which can further improve the encoding and decoding performance in some scenarios.

[0378] ​The terminal device can first perform frequency domain transformation on the auxiliary image to obtain an auxiliary image frequency graph; perform frequency domain transformation on the de-redundancy image to obtain a de-redundancy image frequency graph; then reconstruct the de-redundancy image frequency graph according to the auxiliary image frequency graph to obtain a reconstructed image frequency graph; and finally perform inverse frequency domain transformation on the reconstructed image frequency graph to obtain a reconstructed image.

[0379] That is, the terminal device performs the adjustment in the frequency domain, instead of applying different adjustment functions to different regions of the same image, but applying different adjustment functions to different frequency details of the image (performing blocking and adjusting by block in the frequency domain, which is equivalent to adjusting by frequency level in the original image domain). Although there is a difference in principle, the adjustment in the frequency domain can also achieve the de-redundancy effect by using the information of the rendered intermediate image. Moreover, adjusting by frequency for details to be encoded, since the signal of any frequency affects the pixel value of the entire image, the image generated by the terminal device can have better spatial continuity, which can further improve the coding performance in some scenarios.

[0380] The difference between the embodiment and the embodiment shown in the above Figure 12 The embodiment adds the steps of frequency domain transformation and inverse transformation. The auxiliary image and the image to be encoded are respectively converted into frequency graphs by frequency domain transformation operation (for example, discrete cosine transformation, discrete sine transformation, etc.) at the cloud side, to obtain a de-redundancy image frequency graph, and then the de-redundancy image frequency graph is inverse transformed back to the original image domain. The auxiliary image and the de-redundancy image are respectively converted into frequency graphs by frequency domain transformation operation (for example, discrete cosine transformation, discrete sine transformation, etc.) at the terminal side, to obtain a reconstructed image frequency graph, and then the reconstructed image frequency graph is inverse transformed back to the original image domain, to obtain a final reconstructed image.

[0381] In addition to the above difference, the other steps of the embodiment can refer to the related steps in the embodiment shown in the above Figure 12 , which will not be described here.

[0382] The embodiment performs the adjustment in the frequency domain, instead of applying different adjustment functions to different regions of the same image, but applying different adjustment functions to different frequency details of the image (performing blocking and adjusting by block in the frequency domain, which is equivalent to adjusting by frequency level in the original image domain). Although there is a difference in principle, the embodiment can also achieve the de-redundancy effect by using the information of the rendered intermediate image. Compared with Figure 12 adjusting details by region in the embodiment shown in the above, the embodiment adjusts details by frequency, since the signal of any frequency affects the pixel value of the entire image, the image generated by the terminal device can have better spatial continuity, which can further improve the coding performance in some scenarios.

[0383] Figure 16This is a schematic diagram of the structure of the image encoding device 1600 according to an embodiment of this application, as shown below. Figure 16 As shown, the image encoding device 1600 of this embodiment can be applied to an online server. The image encoding device 1600 may include: a preprocessing module 1601, a redundancy removal module 1602, and an encoding module 1603. Wherein,

[0384] The preprocessing module 1601 is used to render the 3D scene to obtain the image to be encoded and the intermediate rendering results during the rendering process; to preprocess the intermediate rendering results to obtain an auxiliary image; the redundancy removal module 1602 is used to perform redundancy removal processing on the image to be encoded according to the auxiliary image to obtain a redundancy-removed image; and the encoding module 1603 is used to encode the redundancy-removed image to obtain a bitstream.

[0385] In one embodiment, the preprocessing module 1601 is specifically used to preprocess the intermediate rendering results, including at least one of the following processes: anti-aliasing, feature extraction, or high-pass filtering.

[0386] In one embodiment, the anti-aliasing process includes one or more of temporal anti-aliasing, multisampling anti-aliasing, and blurring.

[0387] In one embodiment, the anti-aliasing process is consistent with the anti-aliasing process used in the process of rendering the 3D scene to obtain the image to be encoded.

[0388] In one embodiment, the preprocessing module 1601 is specifically used to obtain a low-frequency map based on the intermediate rendering result; and to extract pixel values ​​that meet preset conditions based on the intermediate rendering result and the low-frequency map to obtain at least one feature map.

[0389] In one embodiment, the preprocessing module 1601 is specifically used to filter the intermediate rendering result to obtain high-frequency information in the intermediate rendering result.

[0390] In one embodiment, the preprocessing module 1601 is specifically used to obtain a low-frequency map based on the intermediate rendering result; and to obtain the high-frequency information by subtracting the intermediate rendering result from the low-frequency map.

[0391] In one embodiment, the preprocessing module 1601 is specifically used to divide the intermediate rendering result to obtain multiple patches to be processed; obtain multiple thumbnails based on the multiple patches to be processed, the thumbnails being obtained based on the pixel mean of the corresponding patches to be processed; and obtain the low-frequency image based on the multiple thumbnails, the low-frequency image being obtained based on the pixel values ​​of the multiple thumbnails.

[0392] In an embodiment, when the rendering intermediate result comprises a normal map, the pre-processing module 1601 is further configured to perform normal processing on the normal map.

[0393] In an embodiment, the pre-processing module 1601 is specifically configured to obtain a light direction and a light intensity of any pixel in the normal map; and obtain a processed pixel value of the pixel according to a normal direction of the pixel in the normal map, the light direction and the light intensity.

[0394] In an embodiment, the pre-processing module 1601 is further configured to multiply the processed pixel value of any pixel in the normal map with a pixel value at a corresponding position in an albedo map.

[0395] In an embodiment, the bitstream further comprises the light direction and the light intensity.

[0396] In an embodiment, the de-redundancy module 1602 is specifically configured to obtain an adjusted auxiliary image, the adjusted auxiliary image being closer to the image to be encoded than the auxiliary image; and perform de-redundancy processing on the adjusted auxiliary image and the image to be encoded to obtain the de-redundancy image.

[0397] In an embodiment, the de-redundancy module 1602 is specifically configured to obtain at least one set of adjustment parameters according to the auxiliary image and the image to be encoded; construct at least one adjustment function according to the at least one set of adjustment parameters; and adjust pixels of a corresponding auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image.

[0398] In an embodiment, the de-redundancy module 1602 is specifically configured to divide the auxiliary image into a plurality of auxiliary image blocks; for any one auxiliary image block, obtain at least one set of adjustment parameters according to the auxiliary image block and a corresponding block in the image to be encoded; construct at least one adjustment function according to the at least one set of adjustment parameters; and adjust pixels of a corresponding auxiliary image block according to the at least one adjustment function to obtain an adjusted auxiliary image block corresponding to the auxiliary image block.

[0399] The apparatus of the embodiment can be used to perform the method of the embodiment. Figure 4 The technical solutions of the method embodiment have similar implementation principles and technical effects, which will not be described herein.

[0400] Figure 17 FIG. 17 is a structural schematic diagram of an image decoding apparatus 1700 according to an embodiment of the present application. Figure 17As shown, the image decoding apparatus 1700 of the present embodiment can be applied to the terminal device described above. The image decoding apparatus 1700 can include a receiving module 1701, a decoding module 1702, a preprocessing module 1703, and a reconstruction module 1704. Wherein,

[0401] The receiving module 1701 is configured to receive a bitstream; the decoding module 1702 is configured to obtain a de-redundancy image according to the bitstream; the preprocessing module 1703 is configured to obtain a rendering intermediate result; and the rendering intermediate result is preprocessed to obtain an auxiliary image; and the reconstruction module 1704 is configured to obtain a reconstructed image according to the auxiliary image and the de-redundancy image.

[0402] In one embodiment, the preprocessing performed by the preprocessing module 1701 on the rendering intermediate result includes at least one of the following: anti-aliasing processing, feature extraction, or high-pass filtering.

[0403] In one embodiment, the anti-aliasing processing includes one or more of the following: time-domain anti-aliasing, multi-sampling anti-aliasing, or blur processing.

[0404] In one embodiment, the preprocessing module 1701 is specifically configured to obtain a low-frequency map according to the rendering intermediate result; and extract pixel values that meet a preset condition from the rendering intermediate result and the low-frequency map to obtain at least one feature map.

[0405] In one embodiment, the preprocessing module 1701 is specifically configured to filter the rendering intermediate result to obtain high-frequency information in the rendering intermediate result.

[0406] In one embodiment, the preprocessing module 1701 is specifically configured to obtain a low-frequency map according to the rendering intermediate result; and obtain the high-frequency information by subtracting the low-frequency map from the rendering intermediate result.

[0407] In one embodiment, the preprocessing module 1701 is specifically configured to divide the rendering intermediate result to obtain a plurality of to-be-processed tiles; obtain a plurality of thumbnails according to the plurality of to-be-processed tiles, the thumbnail being obtained based on a pixel mean value of a corresponding to-be-processed tile; and obtain the low-frequency map based on an interpolation of pixel values of the plurality of thumbnails.

[0408] In one embodiment, when the rendering intermediate result includes a normal map, the preprocessing module 1701 is further configured to perform normal processing on the normal map.

[0409] In an embodiment, the preprocessing module 1701 is specifically configured to obtain the light direction and the light intensity of any pixel in the normal map; and obtain the processed pixel value of the pixel according to the normal direction of any pixel in the normal map, the light direction and the light intensity.

[0410] In an embodiment, the preprocessing module 1701 is further configured to multiply the processed pixel value of any pixel in the normal map by the pixel value at the corresponding position in the albedo map.

[0411] In an embodiment, the preprocessing module 1701 is specifically configured to obtain the light direction and the light intensity of any pixel in the normal map from the bitstream.

[0412] In an embodiment, the reconstruction module 1704 is specifically configured to obtain an adjusted auxiliary image, the adjusted auxiliary image being closer to the reconstructed image than the auxiliary image; and reconstruct the adjusted auxiliary image and the de-redundancy image to obtain the reconstructed image.

[0413] In an embodiment, the bitstream further includes at least one set of adjustment parameters; and the reconstruction module 1704 is specifically configured to obtain the at least one set of adjustment parameters from the bitstream, construct at least one adjustment function according to the at least one set of adjustment parameters, and adjust the pixels of the corresponding auxiliary image according to the at least one adjustment function to obtain the adjusted auxiliary image.

[0414] In an embodiment, the bitstream further includes at least one set of adjustment parameters; and the reconstruction module 1704 is specifically configured to obtain the at least one set of adjustment parameters from the bitstream, any set of adjustment parameters corresponding to an auxiliary image block, the auxiliary image including a plurality of auxiliary image blocks, construct at least one adjustment function according to the at least one set of adjustment parameters, adjust the pixels of the corresponding auxiliary image block according to the at least one adjustment function to obtain an adjusted auxiliary image block corresponding to the auxiliary image block, and obtain the adjusted auxiliary image according to a plurality of the adjusted auxiliary image blocks.

[0415] The apparatus of the embodiment can be used to execute the method of the embodiment. Figure 9 The technical solutions of the method embodiment are similar in principle and technical effects to the apparatus embodiment, and thus are not described herein.

[0416] Figure 18 A schematic block diagram of an apparatus of an embodiment of the present application is shown. As shown in Figure 18 The apparatus 1800 can include a processor 1801 and a transceiver / transmitting / receiving pin 1802, and optionally further include a memory 1803.

[0417] The various components of the apparatus 1800 are coupled together by a bus system 1804, which can include a data bus, a power bus, a control bus, and a state line. For the sake of clarity, the various buses are illustrated in Figure 18 as the bus 1804. The bus 1804 can include other buses as

[0418] Optionally, the memory 1803 can be configured to store instructions of the above method embodiments. The processor 1801 can be configured to execute the instructions in the memory 1803, and control the receiving pin to receive the signal and the sending pin to send the signal.

[0419] The apparatus 1800 can be a device or a chip of the device in the above method embodiments.

[0420] All relevant content of each step of the above method embodiments can be cited to the function description of the corresponding function module, and will not be repeated here.

[0421] In the implementation process, each step of the above method embodiments can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software. The processor can be a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware coding executed by the processor, or executed by a combination of hardware and software modules in the coding processor. The software module can be located in the random access memory, the flash memory, the read-only memory, the programmable read-only memory or the electrically erasable programmable memory, the register or other mature storage medium in the art. The storage medium is located in the storage memory, and the processor reads the information in the storage memory, and combines the hardware to complete the steps of the above method.

[0422] The memory mentioned in each of the above embodiments can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct Rambus RAM (DR RAM). It should be noted that the memory of the system and method described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0423] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0424] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0425] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. The division of the units is merely logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0426] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0427] In addition, each functional unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can be physically present separately, or two or more units can be integrated into one unit.

[0428] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various program codes that can be stored in the medium.

[0429] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image encoding method, characterized in that, include: Render the 3D scene to obtain the image to be encoded and the intermediate rendering results during the rendering process; The intermediate rendering results are preprocessed to obtain auxiliary images; The image to be encoded is subjected to redundancy removal processing based on the auxiliary image to obtain a redundancy-removed image; The deredundant image is encoded to obtain a bitstream.

2. The method according to claim 1, characterized in that, The preprocessing includes at least one of the following processes: Anti-aliasing, feature extraction, or high-pass filtering.

3. The method according to claim 2, characterized in that, The anti-aliasing process includes one or more of the following: temporal anti-aliasing, multisampling anti-aliasing, and fuzzing.

4. The method according to claim 2 or 3, characterized in that, The anti-aliasing process is the same as the anti-aliasing process used in the process of rendering the three-dimensional scene to obtain the image to be encoded.

5. The method according to any one of claims 2-4, characterized in that, The feature extraction includes: The low-frequency map is obtained based on the intermediate rendering results; Based on the intermediate rendering results and the low-frequency map, pixel values ​​that meet preset conditions are extracted to obtain at least one feature map.

6. The method according to any one of claims 2-5, characterized in that, The high-pass filter includes: The intermediate rendering results are filtered to obtain high-frequency information in the intermediate rendering results.

7. The method according to claim 6, characterized in that, The step of filtering the intermediate rendering results to obtain high-frequency information in the intermediate rendering results includes: The low-frequency map is obtained based on the intermediate rendering results; The high-frequency information is obtained by subtracting the intermediate rendering result from the low-frequency map.

8. The method according to claim 5 or 7, characterized in that, The step of obtaining the low-frequency map based on the intermediate rendering results includes: The intermediate rendering results are divided into multiple unprocessed image blocks; Multiple thumbnails are obtained based on the multiple image patches to be processed, and the thumbnails are obtained based on the pixel average of the corresponding image patches to be processed; The low-frequency map is obtained based on the plurality of thumbnails, wherein the low-frequency map is obtained by interpolation of the pixel values ​​of the plurality of thumbnails.

9. The method according to any one of claims 2-8, characterized in that, When the intermediate rendering result includes a normal map, the preprocessing further includes normal processing of the normal map.

10. The method according to claim 9, characterized in that, The normal processing includes: Obtain the illumination direction and illumination intensity of any pixel in the normal map; The processed pixel value of the pixel is obtained based on the normal direction of any pixel in the normal map, the illumination direction, and the illumination intensity.

11. The method according to claim 10, characterized in that, The normal processing also includes: Multiply the processed pixel value of any pixel in the normal map by the pixel value at the corresponding position in the albedo map.

12. The method according to claim 10 or 11, characterized in that, The bitstream also includes the illumination direction and the illumination intensity.

13. The method according to any one of claims 1-12, characterized in that, The step of performing redundancy removal processing on the image to be encoded based on the auxiliary image to obtain a redundancy-removed image includes: Obtain an adjusted auxiliary image, which is closer to the image to be encoded than the original auxiliary image; The adjusted auxiliary image and the image to be encoded are used to perform redundancy removal processing to obtain the redundancy-removed image.

14. The method according to claim 13, characterized in that, The process of acquiring the adjusted auxiliary image includes: At least one set of adjustment parameters is obtained based on the auxiliary image and the image to be encoded; Construct at least one adjustment function based on the at least one set of adjustment parameters; The pixels of the corresponding auxiliary image are adjusted according to the at least one adjustment function to obtain the adjusted auxiliary image.

15. The method according to claim 13, characterized in that, The process of acquiring the adjusted auxiliary image includes: The auxiliary image is divided into blocks to obtain multiple auxiliary image blocks; For any auxiliary image block, at least one set of adjustment parameters is obtained based on the auxiliary image block and the corresponding block in the image to be encoded; Construct at least one adjustment function based on the at least one set of adjustment parameters; The pixels of the corresponding auxiliary image blocks are adjusted according to the at least one adjustment function to obtain the adjusted auxiliary image blocks corresponding to the auxiliary image blocks.

16. An image decoding method, characterized in that, include: Receive the bitstream; Obtain the deredundant image based on the bitstream; Obtain intermediate rendering results; The intermediate rendering results are preprocessed to obtain auxiliary images; The reconstructed image is obtained based on the auxiliary image and the deredundancy-reduced image.

17. The method according to claim 16, characterized in that, The preprocessing includes at least one of the following processes: Anti-aliasing, feature extraction, or high-pass filtering.

18. The method according to claim 17, characterized in that, The anti-aliasing process includes one or more of the following: temporal anti-aliasing, multisampling anti-aliasing, and fuzzing.

19. The method according to claim 17 or 18, characterized in that, The feature extraction includes: The low-frequency map is obtained based on the intermediate rendering results; Based on the intermediate rendering results and the low-frequency map, pixel values ​​that meet preset conditions are extracted to obtain at least one feature map.

20. The method according to any one of claims 17-19, characterized in that, The high-pass filter includes: The intermediate rendering results are filtered to obtain high-frequency information in the intermediate rendering results.

21. The method according to claim 20, characterized in that, The step of filtering the intermediate rendering results to obtain high-frequency information in the intermediate rendering results includes: The low-frequency map is obtained based on the intermediate rendering results; The high-frequency information is obtained by subtracting the intermediate rendering result from the low-frequency map.

22. The method according to claim 19 or 21, characterized in that, The step of obtaining the low-frequency map based on the intermediate rendering results includes: The intermediate rendering results are divided into multiple unprocessed image blocks; Multiple thumbnails are obtained based on the multiple image patches to be processed, and the thumbnails are obtained based on the pixel average of the corresponding image patches to be processed; The low-frequency map is obtained based on the plurality of thumbnails, wherein the low-frequency map is obtained by interpolation of the pixel values ​​of the plurality of thumbnails.

23. The method according to any one of claims 17-22, characterized in that, When the intermediate rendering result includes a normal map, the preprocessing further includes normal processing of the normal map.

24. The method according to claim 23, characterized in that, The normal processing includes: Obtain the illumination direction and illumination intensity of any pixel in the normal map; The processed pixel value of the pixel is obtained based on the normal direction of any pixel in the normal map, the illumination direction, and the illumination intensity.

25. The method according to claim 24, characterized in that, The normal processing also includes: Multiply the processed pixel value of any pixel in the normal map by the pixel value at the corresponding position in the albedo map.

26. The method according to claim 24 or 25, characterized in that, The step of obtaining the illumination direction and illumination intensity of any pixel in the normal map includes: Obtain the illumination direction and illumination intensity of any pixel in the normal map from the bitstream.

27. The method according to any one of claims 16-26, characterized in that, The step of obtaining the reconstructed image based on the auxiliary image and the deredundancy-reduced image includes: Obtain an adjusted auxiliary image, which is closer to the reconstructed image than the original auxiliary image; The reconstructed image is obtained by reconstructing the image based on the adjusted auxiliary image and the deredundant image.

28. The method according to claim 27, characterized in that, The bitstream also includes at least one set of adjustment parameters; The process of acquiring the adjusted auxiliary image includes: The at least one set of adjustment parameters is obtained based on the bitstream; Construct at least one adjustment function based on the at least one set of adjustment parameters; The pixels of the corresponding auxiliary image are adjusted according to the at least one adjustment function to obtain the adjusted auxiliary image.

29. The method according to claim 27, characterized in that, The bitstream also includes at least one set of adjustment parameters; The process of acquiring the adjusted auxiliary image includes: The at least one set of adjustment parameters is obtained according to the bitstream, and any set of adjustment parameters corresponds to an auxiliary image block. The auxiliary image includes multiple auxiliary image blocks. Construct at least one adjustment function based on the at least one set of adjustment parameters; The pixels of the corresponding auxiliary image blocks are adjusted according to the at least one adjustment function to obtain the adjusted auxiliary image blocks corresponding to the auxiliary image blocks; The adjusted auxiliary image is obtained from multiple adjusted auxiliary image blocks.

30. An edge-cloud collaborative system, characterized in that, include: Servers and terminal devices; among which... The server is configured to render a 3D scene to obtain an image to be encoded and intermediate rendering results during the rendering process; preprocess the intermediate rendering results to obtain a first auxiliary image; perform redundancy removal processing on the image to be encoded based on the first auxiliary image to obtain a deredundancy image; encode the deredundancy image to obtain a bitstream; and send the bitstream. The terminal device is configured to receive the bitstream; decode the bitstream to obtain a decoding result, the decoding result including the deredundancy image; acquire a second auxiliary image; and reconstruct based on the deredundancy image and the second auxiliary image to obtain a reconstructed image.

31. The method according to claim 30, characterized in that, The resolution of the first auxiliary image is different from that of the second auxiliary image; and / or, the content of the first auxiliary image is different from that of the second auxiliary image.

32. The system according to claim 30 or 31, characterized in that, The resolution of the second auxiliary image is higher than that of the first auxiliary image.

33. A device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-29.

34. A computer-readable storage medium, characterized in that, Includes a computer program, which, when executed on the device, causes the device to perform the method of any one of claims 1-29.

35. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a device, causes the device to perform the method of any one of claims 1-29.

Images